SYSTEMS AND METHODS FOR INTELLIGENT MEDICAL EDITORS

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to India Provisional Patent Application No. 20/234,106697 filed on Oct. 6, 2024, entitled “SYSTEMS AND METHODS FOR INTELLIGENT MEDICAL EDITORS”. The entirety of the aforementioned application is incorporated by reference herein.

TECHNICAL FIELD

The subject matter disclosed herein relates generally to healthcare technology, specifically for generating and editing of medical reports and records using artificial intelligence (AI) assisted technology.

BACKGROUND

Medical reports, documents, and records are an important part of healthcare around the world. These can include detailed, doctor generated reports describing patient encounters, radiology reports, diagnostic reports, department reports, clinical notes, staff action orders, lists of chronic health conditions, medication lists, imaging results, and the like. Information in the medical reports, documents, and records are typically aggregated for a patient over time in the form an electronic medical record (EMR) for the patient and used to make important decisions affecting patient care.

An EMR or electronic health record (EHR) is a digital version of a patient's paper chart that typically contains their medical and treatment history. EMRs/EHRs are primarily used by healthcare providers (e.g., doctors, nurses, and other medical staff) to store, manage, and retrieve patients' health information electronically. Unlike traditional paper records, EMRs/EHRs offer the ability to track data over time, streamline workflows, and improve the overall management of patient care.

Healthcare professionals (e.g., physicians, nurses, and other healthcare professionals) are generally required to generate medical reports or documentation describing patient encounters and procedures for inclusion in patients' EMRs/EHRs. This requirement serves multiple important purposes, including legal, clinical, and administrative needs. These reports play a crucial role in ensuring quality care, protecting both patients and physicians, meeting regulatory and insurance requirements, and enabling effective communication among healthcare providers. Failing to generate appropriate documentation can lead to legal consequences, poor patient outcomes, and complications in medical billing. The specifics of what needs to be documented and how detailed the reports should be may vary depending on the healthcare setting, type of medical practice, and local regulations, but documentation of patient encounters is an essential part of modern medical practice.

Medical professionals face several challenges when generating and interpreting medical reports, as these documents must accurately reflect the patient's health status, provide clear diagnostic information, and often serve as a basis for critical medical decisions. Physicians, especially in busy hospital settings, often have numerous patients to manage, which limits the time they can dedicate to generating comprehensive medical reports. This can lead to rushed or incomplete documentation. The high demand for generating detailed reports, along with other clinical responsibilities, can contribute to physician burnout, reducing the quality of reports. Doctors must balance time spent on patient care with report writing. This can be particularly challenging in specialties like radiology or emergency medicine, where prompt reporting is essential for patient outcomes. Systems and methods are needed to speed up the time users take to generate, edit and update medical materials and save money for the stakeholders involved.

Ensuring the correctness, completeness and clarity medical reports is paramount to improvement of clinical outcomes. Imprecise and incomplete EMRs, and other medical materials, adversely affect the speed and quality of all the subsequent clinical decisions such as diagnosis/treatment decisions, clinical trial selection, referencing guidelines and case studies. Ensuring that all relevant patient data (e.g., clinical history, diagnostic test results, medication lists) is captured in medical reports correctly is critical. Human-made errors frequently happen when writing medical reports and editing medical records. Information may be missed, instead of captured. Errors or omissions in reports can lead to misdiagnosis or inappropriate treatment. Spelling mistakes can change the meaning of information and affect patient care—for example affecting a dosage of medication for a patient. If reports contain vague or unclear medical terminology, other healthcare providers might misinterpret the findings, leading to poor clinical outcomes. Ambiguity may include poor phasing, or different types of phrasing for the same issue across medical professions in different regions or languages. In addition, different specialties may use distinct jargon or clinical perspectives when generating medical reports. For example, a radiologist's interpretation of imaging findings might not align with a surgeon's clinical priorities. Poorly communicated findings between departments can result in inappropriate treatment. For instance, an unclear radiology report might lead to unnecessary surgery, or an ambiguous pathology report could delay cancer treatment. And, of course, information may just be wrong instead of right. Systems and methods are needed to improve such issues within medical materials.

Medical reports may include data from various sources (e.g., lab results, imaging reports, specialist opinions), which can be inconsistent or outdated, leading to confusion or conflicting information. Contradictions inside a record or between different records for a given patient may exist instead of correctness. Further, inconsistencies of information between the medical materials and medical standards or guidelines may lead to legal or medical issues. Medical reports are legal documents, and any inaccuracies or omissions could expose doctors to malpractice claims. Physicians are often cautious about how they phrase uncertain diagnoses, fearing legal repercussions. In an effort to protect themselves legally, doctors may include excessive details or hedge their conclusions in a way that diminishes the clarity and utility of the report for other healthcare providers. Clear documentation guidelines from healthcare institutions help doctors understand their obligations and reduce the fear of litigation. However, medical guidelines and diagnostic criteria frequently evolve. Doctors must stay current with these changes to ensure their reports meet updated standards of care. This can be especially challenging in fast-moving fields like oncology or infectious disease, where new therapies or diagnostic criteria emerge regularly. In some cases, different medical organizations may issue conflicting guidelines, making it difficult for doctors to know which criteria to apply when generating reports. Systems and methods are needed to help generate and ensure medical material are accurate, comprehensive and comply with industry standard medical guidelines.

The complexity of medical data makes medical report generation and interpretation a particularly challenging task. Physicians must distill complex clinical information, lab results, and diagnostic images into a concise and understandable report. For example, translating sophisticated imaging findings into actionable clinical information can be difficult, especially for non-specialist readers. With the advent of EMRs/EHRs and big data in medicine, doctors are often inundated with large amounts of patient information. Extracting the most relevant data and presenting it in an organized manner can be overwhelming.

Physicians may struggle to efficiently integrate patient data from different sources or manually input findings into the system. Medical professionals these days also have to deal with many computer and information technology (IT) systems along the patient journey. In addition, a medical professional may be required to use multiple tools on different platforms during the course of a patient journey and manually work with IT systems, which include EMR and EHR reporting systems. These varying IT systems may have gaps in compatibility and increase the burden on a medical professional, especially if the professional needs to provide a comprehensive multi-modal patient analysis. Furthermore, EMR/EHR reporting systems, while useful, often have cumbersome interfaces that make report generation time-consuming. Systems and methods are needed to improve electronic medical reporting systems both in terms of compatibility and usability.

The above-described background relating to issues associated with medical reporting are intended to provide a contextual overview of some current issues and is not intended to be exhaustive. Other contextual information may become further apparent upon review of the following detailed description.

SUMMARY

This summary introduces concepts that are described in more detail in the detailed description. It should not be used to identify essential features of the claimed subject matter, nor to limit the scope of the claimed subject matter.

In an embodiment, a system is provided that comprises a processor; and a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations. These operations comprise identifying a textual entities in a medical report associated with a patient in association with receiving user input entering at least some of the textual entities into the report via a graphical user interface, and determining edit recommendations related to one or more of the textual entities, where the edit recommendations are generated by an artificial intelligence model (e.g., a large language model (LLM), and wherein the edit recommendation relate to possible errors or improvements based on information comprising medical history data of the patient, one or more clinical guidelines applicable to the report, and one or more medical ontologies. These operations further comprise providing feedback information regarding the edit recommendations via the graphical user interface, receiving user feedback regarding selections related to the edit recommendations, and retraining the artificial intelligence model using the report and the user feedback as a training data, resulting in an updated version of the artificial intelligence model.

In various implementations, the errors or improvements based on one or more of the following: the one or more textual entities being clinically incorrect; the one or more textual entities being incompatible with the one or more clinical guidelines; the one or more textual entities being incompatible with the one or more medical ontologies; the one or more textual entities being inconsistent or incompatible with the medical history data of the patient; the one or more textual entities being clinically inconsistent with one or more other textual entities included in the report; and detected differences between the report and a related synoptic reporting standard related to a type of the report.

In some implementations, the feedback information comprises auxiliary information regarding a basis of the errors or improvements. Thet auxiliary information can comprise one or more links to a relevant portion of the one or more clinical guidelines or the medical history data.

In some implementations, the operations further comprise generating the textual content included in the report based on the medical history data and using a generative large language model and associating one or more visual indicators with the textual content indicating a measure of confidence pertaining to an accuracy level of the textual content.

The operations can further comprise receiving additional user feedback regarding the textual content, the additional user feedback comprising an acceptance of the textual content, a rejection of the textual content or a revision to the textual content and retraining the generative large language model using the report and the additional user feedback as training data.

In some embodiments, elements described in connection with the disclosed systems can be embodied in different forms such as a computer-implemented method, a computer program product (e.g., a non-transitory machine-readable medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations described with respect to the disclosed systems) or another form.

BRIEF DESCRIPTION OF DRAWINGS

Numerous aspects, implementations, objects and advantages of the present invention will be apparent upon consideration of the following detailed description, taken in conjunction with the accompanying drawings, in which like reference characters refer to like parts throughout, and in which:

FIG. 1 shows stages in a patient journey, according to an embodiment.

FIG. 2A shows a hardware device for a healthcare environment, according to an embodiment.

FIG. 2B shows a block diagram for a computed tomography hardware and software system, according to an embodiment.

FIG. 3 shows a block diagram for an intelligent medical editor, according to an embodiment.

FIG. 4 shows a flow diagram for an intelligent medical editor, according to an embodiment.

FIG. 5 shows a natural language processing (NLP) model update module, according to an embodiment.

FIG. 6 shows a natural language generation (NLG) module with a natural word prediction (NWP) module, according to an embodiment.

FIG. 7 shows a block diagram supporting natural report generation, according to an embodiment.

FIG. 8A shows a local model for language fine tuning, according to an embodiment.

FIG. 8B shows details for a language fine tuning module, according to an embodiment.

FIG. 9 shows a clinical information extraction module, according to an embodiment.

FIG. 10 shows an edit recommendation module, according to an embodiment.

FIG. 11 shows a contradiction and incompleteness detection module, according to an embodiment.

FIG. 12 shows a clinical care guidelines deviation flow, according to an embodiment.

FIG. 13 shows a user interface with suggested edits for a digital document, according to an embodiment.

FIG. 14 shows details for an enriched information overlay module, according to an embodiment.

FIG. 15 shows a comparison between a standard report and a suggestions-supported report, according to an embodiment.

FIG. 16 shows a user interface with suggested edits to a digital document, according to an embodiment.

FIG. 17 shows a user interface with medical corrections in a digital document, according to an embodiment.

FIG. 18 shows a block diagram related to edit recommendations leveraging predictive and generative models, according to an embodiment.

FIG. 19 shows a flow chart and block diagram related to patient profiling, according to an embodiment.

FIG. 20A shows a flow chart for updating a medical report based on selected suggested edits, according to an embodiment.

FIG. 20B shows a flow diagram of an example computer-implemented method for generating a medical report using an intelligent medical reporting tool, according to an embodiment;

FIG. 21 shows a schematic block diagram of a sample-computing environment, according to an embodiment.

FIG. 22 shows a schematic block diagram of another sample-computing environment, according to an embodiment.

FIG. 23 shows a schematic block diagram of another sample-computing environment, according to an embodiment.

FIG. 24 shows a schematic block diagram of another sample-computing environment, according to an embodiment.

FIG. 25 shows a schematic block diagram of another sample-computing environment, according to an embodiment.

FIG. 26 shows a schematic block diagram illustrating a suitable operating environment, according to an embodiment.

DETAILED DESCRIPTION

The following detailed description is merely illustrative and is not intended to limit embodiments and/or application or uses of embodiments. Furthermore, there is no intention to be bound by any expressed or implied information presented in the preceding Background section, Summary section or in the Detailed Description section.

In an embodiment, systems, computer-implemented methods, apparatus and/or computer program products are described that facilitate efficiently and accurately generating and editing electronic medical reports by medical professional throughout the patient journey and numerous downstream analyses leveraging, but not limited to, standardized EMRs, clinician interactions and multi-modal patient history. The disclosed techniques solve multiple problems described and indicated in the Background section associated with medical reporting using artificial intelligence (AI) techniques to improve electronic medical report generation efficiency, correctness, completeness and clarity.

In various embodiments, the disclosed techniques use AI to facilitate efficiently and accurately generating medical reports by medical professionals that comply with applicable medical guidelines and ontologies using an electronic reporting tool. For example, the electronic reporting tool can provide report templates tailored to different types of medical reports, clinical indications, procedures, medical domains, medical departments, organizations, and the like, that can include structured data fields for required information to be included in the respective reports and/or free-text boxes via which the user can enter free text describing relevant information for inclusion in the report. Free text refers to any unstructured, natural language input that is not constrained by predefined fields, templates, or formats. In contrast to structured data, where information is entered into specific fields (like a drop-down menu or a checkbox), free text allows users to input information in their own words, without constraints.

In addition to allowing the user to enter information manually, the reporting tool can integrate AI technology that provides for automatically integrating relevant and required (in accordance with applicable clinical guidelines) clinical and non-clinical information associated with the patient as included in an existing electronic medical record for the patient (e.g., an EMR/EHR,) and/or various disparate medical information sources/systems (e.g., multimodal medical record databases and information systems, such as a radiology information system (RIS), a medical image archiving database, a laboratory information system, or the like) at the time of report generation. In other words, based on the type of the report, predefined structured data fields included in the report, predefined information prompts included in the report, a clinical guideline/ontology applicable to the report and learned information (e.g., as learned over time using machine learning (ML)) about what information should be included in the type of the report, the using AI technology (e.g., afforded by Large Language Models (LLMs)), the reporting tool can autofill as much information into the report as available in or determined from existing medical records for the patient.

In some embodiments, using generative AI techniques (e.g., afforded by Large Language Models (LLMs), the disclosed reporting tool can further generate structured, relevant and required content for inclusion in the report that ensures compliance with the applicable clinical guidelines and standardized medical ontologies in a concise manner that facilitates efficient and consistent interpretation by disparate healthcare professionals and organizations. For example, given a prompt in the report, such as a prompt indicating a particular category of information to be described in free-text box (e.g., a prompt requesting a summary of the patient encounter, a prompt requesting a summary of patient symptoms, a prompt requesting a recommended treatment plan, etc.), existing medical records for the patient, and applicable clinical guidelines and ontologies for the report type, a pre-trained LLM can automatically generate the corresponding content.

In association with automatically integrating and generating relevant information and receiving information entered by the user, the disclosed AI assisted reporting tool can identify and facilitate correcting reporting issues (e.g., via providing feedback and/or automatically correcting certain issues), including errors attributed to omitted information, ambiguous information, extraneous/irrelevant information and incorrect information, including incorrect information attributed to inconsistencies between information within the report itself, the patient's prior medical history, and applicable clinical guidelines and ontologies. For example, using AI technology, the reporting tool can identify and facilitate correcting spelling and grammatical errors as tailored to the applicable clinical ontologies and guidelines. However, as opposed to merely correcting spelling and grammatical errors, the AI assistance provided by the reporting tool can identify and facilitate correcting conceptual errors as tailored based on the type of the report, the applicable clinical guidelines and ontologies, and the patient based on existing information known about the patient and the patient's medical history.

For example, the reporting tool can identify inconsistences between information entered by the user within the report, such as initial description of the patient's pathology involving the right lung and later referring to the left lung. The reporting tool can also identify inconsistencies between information entered by the user that does not comply with known information about the patient as included previous reports, existing information in the patient's EMR/EHR, medical imaging records, laboratory records, and the like. The reporting tool can also identify inconsistencies between information entered by the user that does not reflect applicable medical guidelines, such as information providing a medical interpretation or diagnosis of the patient that is inaccurate with respect to the applicable clinical guidelines or ontologies given what is known about the patient. In another example, the reporting tool can identify errors attributed to usage of incorrect medical terminology, such as usage of the wrong medical term based on the definition of the term, as provided in a standardized medical ontology or dictionary, and known information about the patient. Similarly, the reporting tool can identify and facilitate correcting usage of ambiguous terms, acronyms, and replace similar terms with standardized terms in accordance with applicable clinical guidelines and ontologies.

The reporting tool can also identify missing information that is required for the report based on the type of the report and an applicable clinical guideline and suggest or automatically enter the missing information. The missing information can be based on a requirement for the report or contextual based on AI generated content and/or user provided content during generation of the report. In an example, if the user provides information describing a particular event that occurred in the patient's treatment and the applicable clinical guideline for the report requires the event to include a date and location of occurrence, the reporting tool can determine that the date and location information is needed at the time of entry of the event information and automatically enter this information (as retrieved in existing medical records for the patient) and/or prompt the user to add this information. The reporting tool can also identify extraneous or irrelevant information that is included in the report based on the type of the report and an applicable clinical guideline and suggest removing or automatically remove the extraneous/irrelevant information. Using generative AI technology, the reporting tool can also rephrase user entered information in a more concise and accurate manner that is consistent with applicable clinical guidelines and ontologies.

In various embodiments, the reporting tool can provide feedback regarding identified errors during report generation that enables the user to correct the errors. The reporting tool can similarly identify and provide feedback regarding errors identified in previously generated reports in association with editing previously generated reports. Such feedback may be in the form of a visual indication such as highlighting, underlying or the like, with distinctive indicators (e.g., different colors, icons, patterns, etc.) that reflect different types of errors. For example, the different types of errors can be predefined and include, but are not limited to, an ambiguity error, a vocabulary error, a grammatical error, a spelling error, a contradiction error, an incompleteness error, a wrong information error, an extraneous/irrelevant information error, and the like. In some embodiments, the feedback information can also include a description of the basis for the error and in some implementations, include a link or hyperlink to relevant information providing the basis for the error. For example, as applied to contradiction errors based on inconsistencies between information included in the report and existing information known about the patient or an applicable clinical guideline or ontology, the feedback information can include a link to the corresponding existing information (e.g., a entry in the patient's EMR, a radiology report, an entry in the radiology report, a medical image showing a particular pathology, and the like) or the portion of the applicable clinical guideline/ontology for which the inconsistency is based.

The reporting tool can also provide feedback regarding AI generated content integrated into the report that indicates a measure of confidence in the accuracy of the AI generated content. Such feedback may be in the form of a visual indication such as highlighting, underlying or the like, with distinctive indicators (e.g., different colors, icons, patterns, etc.) that reflect corresponding measures of confidence in the AI generated content (e.g., a red color may be used to indicate a low confidence level, a yellow color may be used to indicate medium confidence level, and a green color may be used to indicate a high confidence level). These types of visual confidence indicators are easier to interpret by medical professionals as opposed to technical confidence metrics used by AI developers (e.g., confidence scores, confidence percentages, Dice scores, etc.). Additionally, or alternatively, such technical confidence metrics can be used. In some implementations, the feedback can also include a link or hyperlink to relevant information providing the basis for which the AI content was generated, such as relevant information included medical records for the patient, including text data, laboratory data, image data, and other types of data), and relevant portions of applicable clinical guidelines and ontologies.

In addition to the benefits attributed to the disclosed reporting tool in terms of improving the efficiency and accuracy of medical reporting by healthcare professionals, the user feedback received from the interactions with the error feedback and the AI generated content feedback is used to improve the AI assistance itself. In this regard, as the user is presented with feedback regarding identified errors and possible errors in AI generated content at the time of report generation and/or editing, the user can review the feedback and accept, reject or revise the report accordingly. In this manner, the user can provide user feedback indicating whether the AI assisted insights are correct (e.g., based on accepting and responding to identified errors and accepting AI generated content), incorrect and why (e.g., based on correcting the AI generated content, disregarding identified errors, editing suggested corrections and AI generated content, and so on), or partially correct and why (e.g., based on correcting the AI generated content, disregarding identified errors, editing suggested corrections and AI generated content, and so on). This user feedback is further used to tailor and improve the accuracy of the AI assistance technology of the reporting tool over time (e.g., via AI model updating/tuning). The improvements are done in a federated way without compromising the patient privacy.

Further, because the user feedback is received from domain experts in the corresponding medical fields generating the reports themselves (e.g., doctors, nurses, and other types of trained healthcare professions), the user feedback is considered highly accurate and relevant and does not require additional review by domain experts prior to usage thereof for AI model updating/tunning. As such, the disclosed techniques can be used to continuously and efficiently (e.g., in terms of resources involved, training data curation costs, and so on) improve the accuracy and specificity of the AI assistance features provided by reporting tool overtime. In addition, in some embodiments, different versions of the one or more AI models (e.g., one or more LLMs and other types of machine learning models) used by the reporting tool to perform error identification and automatically generate content can be tailored to individual medical professionals and their preferences, different types of reports, different medical providers and organizations, different clinical specialties and the like.

Furthermore, in association with automatically generating content included in the report, and/or rephasing, replacing terms and phrases manually entered by the users, the LLMs can be configured to use standardized medical terminology in accordance with the applicable clinical guidelines, standardized ontologies such as UMLS (Unified Medical Language System), SNOMED (Systematized Nomenclature of Medicine) and others, and standardized formats, such as the FHIR (Fast Healthcare Interoperability Resources format) format, the HL7 (Health Level 7) format, the DICOM (Digital imaging and Communications in Medicine) format, and the like. Additionally, or alternatively, the LLMs can generate and integrate metadata with terms and phrases included in the report (e.g., as automatically generated and/or provided manually by the user) that maps the terms and phrases to corresponding concepts (e.g., UMLS codes, International Classification of Diseases (ICD) codes or the like) as defined in the standardized ontologies. This is highly beneficial for downstream AI application processing of the reports as included in the respective patients EMR/EHRs following generation thereof.

In particular, EMRs/EHR typically include a vast amount of unstructured medical information, including text, image data, laboratory data, clinician notes and the like, received from disparate sources that use different reporting formats. Pre-processing of the unstructured medical information is a complex but crucial step that enables the information to be processed by many AI algorithms to generate meaningful insights. For example, the pre-processing involves techniques such as text cleaning, tokenization, entity recognition, and context handling to ensure that free-text clinical notes, lab reports, and other medical data are transformed into structured, interpretable formats. These steps enable downstream AI applications to perform tasks like diagnosis prediction, clinical decision support, and natural language understanding with greater accuracy.

With the disclosed AI assisted reporting tool, this pre-preprocessing is performed at the time of report generating and editing. In this regard, the reports generated via the reporting tool are stored in the patients' EMR/EHRs as already pre-processed and structured. As a result, the corresponding reports do not need to be pre-processed prior to usage by downstream AI applications, and thus significantly improves the accuracy and efficiency (e.g., in terms of processing speed, computing resources used, and associated costs) of such downstream AI applications.

In this regard, a number of technical effects are produced by the systems and methods herein, including identifying edit-candidates (LLM), factual contradiction and incompleteness detection using existing patient data, clinical guidelines and medical ontologies, incompleteness detection and report structure recommendation using synoptic reporting standards, cross-reference recommendation, providing a standard tool for writing medical reports providing the suggesting the alternative words using medical standards vocabulary, and linking using patient data model and clinical guidelines, and re-aligning reports to different templates using (LLMs). Further technical effects produced by the systems and methods herein include automatically identifying missing body parts (due to biological sex or a past surgery where the body part was removed), e.g., a male patient is diagnosed with ovarian cancer, and medical guideline compatibility detection. For example, warnings can be popped up if a treatment decision is taken which is not present in any of the treatment options suggested by care guidelines.

Medical text processing is an important application of natural language processing (NLP) and supervised machine learning is used to process the medical text to extract useful information. Supervised machine learning techniques require annotated text to learn from the data. All the useful words need to be labelled with the most appropriate label out of all the possible class labels. The annotated text needs to be in a format which can represent the word-label relations and other representations based on the NLP task. Some of the common formats include XMI (XML metadata exchange) and CoNLL. The CoNLL format is a text file with one word per line with sentences separated by an empty line.

The time taken and redundancy can be reduced with the help of a smart annotation and editing tool which can help annotate the reports while writing the report itself by providing label suggestions for each word. This requires an AI enabled reporting tool, which needs to be trained with limited data. The tool helps us to get annotated data quickly. The model can be tuned at the end of each report by comparing the predictions with actual annotation selection by doctors there by improving the adaptability of model for specific doctor or type of report.

In an embodiment, an intelligent, AI enabled reporting tool can be very impactful as multiple modules can be integrated into the tool that can provide intelligence based on the domain. This will help medical health professionals get annotated data quickly to perform NLP tasks in different domains. In an embodiment, an AI enabled reporting tool also helps to bring a standard form of reporting and brining similarity in the wide variety of reports. This is beneficial to training the model with lesser data.

In the following detailed description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific examples that may be practiced. These examples are described in sufficient detail to enable one skilled in the art to practice the subject matter, and it is to be understood that other examples may be utilized, and that logical, mechanical, electrical and other changes may be made without departing from the scope of the subject matter of this disclosure. The following detailed description is, therefore, provided to describe an exemplary implementation and not to be taken as limiting on the scope of the subject matter described in this disclosure. Certain features from different aspects of the following description may be combined to form yet new aspects of the subject matter discussed below.

When introducing elements of various embodiments of the present disclosure, the articles “a,” “an,” “the,” and “said” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements.

FIG. 1 shows stages in a patient journey, according to an embodiment. FIG. 1 includes patient journey 100 with steps of information gathering 102, information integration and interpretation 104, working diagnosis 106, treatment not needed 108, treatment planning 110, information gathering 112, information integration and interpretation 114, treatment administration 116, information gathering 118, information integration and interpretation 120, repeat process 122, treatment conclusion 124, and follow-up 126. Patient journey 100 also has patient stages of pre-diagnosis 130, pre-treatment 132, in-treatment 143, and post-treatment 146.

An initial stage of a patient journey is pre-diagnosis 130. This includes the initial information gathering 102 before or during an initial medical appointment. This information may be data, observations, measurements, images, test results, medical opinions, evidence, or other information the patient or initial consult medical provider gathers. In this stage, patients seek care for their symptoms with a provider, often a primary care physician or other medical generalist. And later in patient journey stages, specialists may be consulted. Sometimes a patient directly seeks out a specialist to evaluate their condition, which may or may not shorten their journey depending on their choice of an appropriate specialist. Information gathering 102 may be from a variety of information gathering modes, such as clinical history and interview, physical examinations, diagnostic testing, consultation with experts, and the like.

Various steps in the patient journey involve the generation of medical reports and other type of medical information (e.g., laboratory information, imaging study information, clinical nodes and other types of records) that can be integrated within or linked to the patient's EMR. Over the patient journey, any information reported or gathered for the patient facilitate making clinical decisions regarding downstream care, including treatments, procedures, medications, and so one. For example, information may be integrated in step 104 into an EMR, which may include gathered evidence, reasonings and thought processes, conclusions, prescriptions for further tests, prescriptions for treatment, treatment planning thought process, treatment progress notes, treatment response notes, and follow up notes.

Information generation, integration and interpretation steps 104, 114, and 120 may be enabled by an intelligent medical reporting tool, as described herein throughout. Information can be analyzed by an intelligent medical reporting tool and then presented with suggestions for integration and interpretation to speed up and improve information generation, integration and interpretation steps 104, 114, and 120. Such an intelligent medical reporting tool can provide auto-creation of reports as well as automatic suggestions and edit recommendations. The intelligent medical reporting tool further uses tracked information gathered over the patient's journey to understand the current position and health status (or disease state) in their journey to provide recommendations for integrating in new reports accordingly.

During an initial consultation with a medical provider, an initial working diagnosis 106 is developed, and may be reported and integrated with the patients EMR using by the patient's physician using the intelligent medical reporting tool. Sometimes treatment may not be needed as a medical issue may be resolved on its own, and the patient journey ends at treatment not needed 108. Other times, more information is gathered at information gathering 102. And many times, an initial treatment is planned at treatment planning 110. Treatment planning 110 may be done with the initial medical provider, along with nurses, specialists, and the patient. In various embodiments, the disclosed intelligent medical reporting tool or can suggest treatments based on medical guidelines and previous information gathered/tracked for the patient along their journey.

During treatment planning 110, a medical provider works with a number of different types of information and medical devices to continue to perform information gathering 112. And the information gathered feeds into information integration and interpretation 114. In an embodiment, this is done with a computing device running the intelligent medical reporting tool. These steps may be done in pre-treatment phase 132 of a patient journey.

The in-treatment phase 134 of the patient journey may include treatment administration 116. Treatment administration 116 may include many types of treatment based on the symptoms and underlying medical issue, such as watch and wait, chemotherapy or other drug therapies, radiation therapy, immunotherapy, vaccine therapy, stem cell transplantation, blood transfusion, palliative care, physical therapy, and other medical treatments. A health system may provide a patient with both on-site and follow-up care (prescriptions, physical therapy, counseling, or suggested lifestyle changes).

The in-treatment phase 134 of the patient journey also includes further information gathering 118, information integration and interpretation 120, and potentially repeating the process 122. Repeating the process may be done as needed, because this phase of the diagnostic journey may include involves one or multiple evaluations by specialists. For example, a patient presenting with fever of unknown origin may be referred by their primary care physician to a rheumatologist and infectious disease specialist simultaneously or in sequence. This phase may be expedited in inpatient settings where multi-specialty consultation is available, particularly at academic medical centers.

The post-treatment phase 136 of the patient journey may include treatment conclusion 124 and follow-up 126. Treatment conclusion 124 occurs when a medical condition is at the point of taking a break or being complete with treatment. The patient manages his or her care between clinical visits; meanwhile, the health system fosters engagement between the patient and physician in order to enable the patient to address symptoms and maintain good health. This patient follow up and retention phase is a crucial part of the patient journey that is often forgotten after a patient receives treatment. Getting feedback through follow up 126 is necessary so the system can gather insights into patient experiences and satisfaction. The patient's input can then be used to improve patient journey mapping, which ultimately helps to improve the patient experience, quality of care and process efficiency.

FIG. 2A shows a hardware device for a healthcare environment, according to an embodiment. FIG. 2B shows a block diagram for a computed tomography hardware and software system, according to an embodiment. Such machines may be used as medical imaging devices along a patient journey.

FIGS. 2A and 2B show a computed tomography (CT) imaging system 10 including a gantry 12. Gantry 12 has a rotary member 13. An x-ray source 14 that projects a beam of x-rays 16 through pre-patient collimator 15 toward a detector assembly 18 on the opposite side of the rotary member 13. X-ray source 14 may also be referred to as x-ray tube or x-ray generation component. X-ray source 14 is a type of emissions component. A main bearing may be utilized to attach the rotary member 13 to the stationary structure of the gantry 12. Detector assembly 18 is formed by a plurality of detectors 20 and data acquisition systems (DAS) 22, and can include a post-patient collimator. The plurality of detectors 20 sense the projected x-rays that pass through a subject 24, and DAS 22 converts the data to digital signals for subsequent processing. Each detector 20 produces an analog or digital electrical signal that represents the intensity of an impinging x-ray beam and hence the attenuated beam as it passes through subject 24. During a scan to acquire x-ray projection data, rotary member 13 and the components mounted thereon can rotate about a center of rotation.

Rotation of rotary member 13 and the operation of x-ray source 14 are governed by a control mechanism 26 of CT system 10. Control mechanism 26 can include an x-ray controller 28 and generator 30 that provides power and timing signals to x-ray source 14 and a gantry motor controller 32 that controls the rotational speed and position of rotary member 13. An image reconstructor 34 receives sampled and digitized x-ray data from DAS 22 and performs high speed image reconstruction. The reconstructed image is output to a computer 36 which stores the image in a computer storage device 38.

Computer 36 also receives commands and scanning parameters from an operator via operator console 40 that has some form of operator interface, such as a keyboard, mouse, touch sensitive controller, voice activated controller, or any other suitable input apparatus. Display 42 allows the operator to observe the reconstructed image and other data from computer 36. The operator supplied commands and parameters are used by computer 36 to provide control signals and information to DAS 22, x-ray controller 28, and gantry motor controller 32. In addition, computer 36 operates a table motor controller 44 which controls a motorized table 46 to position subject 24 and gantry 12. Particularly, table 46 moves a subject 24 through a gantry opening 48, or bore, in whole or in part. A coordinate system 50 defines a patient or Z-axis 52 along which subject 24 is moved in and out of opening 48, a gantry circumferential or X-axis 54 along which detector assembly 18 passes, and a Y-axis 56 that passes along a direction from a focal spot of x-ray tube 14 to detector assembly 18.

Such imaging systems provide data, images, and information for the information gathering stage and interpretation along the patient journey. In some embodiments, computer 36 may have an intelligent medical reporting tool that enables the generation of radiology reports. For example, in an embodiment, the intelligent medical reporting tool can provides a CT image along with a radiology report that was generated and/or edited using the intelligent medical reporting tool as disclosed within. Then the radiology report may be further integrated into a larger EMR using a different instance, or the same, intelligent medical reporting tool.

FIG. 3 shows a block diagram for an intelligent medical reporting system 300, according to an embodiment. System 300 can include or correspond to one or more computing devices, machines, virtual machines, computer-executable components, datastores, and the like that may communicatively coupled to one another either directly or via one or more wired or wireless communication frameworks. Aspects of the systems, apparatuses or processes explained in this disclosure can constitute computer-executable or machine-executable component(s) embodied within machine(s), e.g., embodied in one or more computer readable mediums (or media) associated with one or more machines. Such component(s), when executed by the one or more machines, e.g., computer(s), computing device(s), virtual machine(s), etc. can cause the machine(s) to perform the operations described.

In this regard, system 300 various computer-executable or machine-executable components, including includes intelligent medical reporting tool 302, clinical information extraction module 304, enriched information overlay module 306, edit recommendation module 308, usage and feedback logging module 310, contradiction and incompleteness module 312, language model finetuning module 314, natural language understanding (NLU)) model updation module 316, NLU models 320 and clinical language models 330. These computer-executable or machine-executable components can be stored in a least one memory and executed by at least one processor or processing unit to perform the corresponding operations described. Examples of said memory, processing unit and other suitable computer or computing-based elements, can be found with reference to FIG. 2600 (e.g., system memory 2606 and processing unit 2604 respectively), and can be used in connection with implementing one or more the components shown and described in connection with FIG. 3, and/or other figures disclosed herein.

System 300 also include various data sources and/or data structures that can provide/store information that is used by and/or generated by the intelligent medical reporting tool 302, including one or more storage databases 303, medical data structures 318, medical ontologies 322, patient profile and past Dx/Tx data 324, clinical guidelines 326, synoptic reporting standards 328, usage logs 332, and user profile and preferences 334. Intelligent medical reporting system 300 may be an intelligent medical tool, or parts of system 300 may be the intelligent medical reporting tool, in varying embodiments. Individual modules may be within the intelligent medical reporting software or in other connected software programs. They may be separate modules within a larger program, or separate programs linked together, such as by procedure calls. FIG. 3 indicates the functional components of a system 300 including an intelligent medical reporting tool 302.

Intelligent medical reporting tool 302 is an example tool for generating and editing electronic medical records. The medical records can include various types of medical reports generated/obtained over the course of the patient journey, such as physician reports, radiology reports, laboratory reports, diagnostic reports, patient encounter reports, clinical notes, clinical orders and other medical materials that can be included in and/or linked to the patients EMR. In some embodiments, the intelligent medical reporting tool 302 can facilitate efficiently and accurately generating the medical records in a structured format (e.g., in accordance with FHIR, HL7 and others in accordance with applicable clinical guidelines standardized ontologies (e.g., UMLS and others) and previously generated medical records for the patient. Additionally, or alternatively, the intelligent medical reporting tool 302 can be used to review and edit previous generated medical records for the patient.

The intelligent medical reporting tool 302 can provide a user interface that interacts with users, both presenting information to, and receiving information from, users. The intelligent medical reporting tool 302 is connected to one or more data storage databases 303 (e.g., EMR systems, a RIS, and other types of medical information storage systems) where newly generated reports and records can be stored for and/or previously generated reports and records can be stored for facilitating generating the new reports and for updating previously generated medical reports and records. In some embodiments, the intelligent medical reporting tool 302 can send unstructured data and structured data included in the one or more data storage databases 303 to clinical information extraction module 304. The intelligent medical reporting tool 302 may receive enriched information from enriched information overlay module 306, edit recommendations from edit recommendation module 308, and user decisions and feedback from usage and feedback logging module 310. The intelligent medical reporting tool 302 may also send information to these modules based on user interactions. For example, if edit recommendation module 308 provides an edit suggestion and a user selects it, the response to that recommendation may go back to the edit recommendation module 308 for implementing the change as well as being input into usage logs 332 through usage and feedback logging module 310 to better adaptively learn user behavior—both individually to the specific user and generally to all users who had a similar suggestion presented by the intelligent medical reporting tool 302.

In various embodiments, to facilitate generating new medical reports/records or reformatting and/or editing previously generated medical reports, the intelligent medical reporting tool 302 can provide a template record or template report that can be presented to users, pre-populated with data, and then overlaid with suggestions and edit recommendations. For example, the intelligent medical reporting tool 302 can provide report/records templates tailored to different types of medical reports/records, clinical indications, procedures, medical domains, medical departments, organizations, and the like, that can include structured data fields for required information to be included in the respective reports and/or free-text boxes via which the user can enter free text describing relevant information for inclusion in the report. These report/record templates can be included with and/or formatted in accordance with corresponding synaptic reporting standards 328 and clinical guidelines 326.

In addition to allowing the user to enter information manually, the intelligent medical reporting tool 302 can integrate AI technology (e.g., one or more NLU models 320 and/or one or more clinical language models 330) that provides for automatically integrating relevant and required (in accordance with applicable clinical guidelines) clinical and non-clinical information associated with the patient as included in an existing electronic medical record for the patient (e.g., an EMR/EHR,) and/or various disparate medical information sources/systems (e.g., multimodal medical record databases and information systems, such as a radiology information system (RIS), a medical image archiving database, a laboratory information system, or the like) at the time of report/record generation and/or editing. In other words, based on the type of the report, predefined structured data fields included in the report, predefined information prompts included in the report/record, a clinical guideline/ontology applicable to the report/record and learned information (e.g., as learned over time using machine learning) about what information should be included in the type of the report/record, using one or more NLU models 320 and/or clinical language models 330, the intelligent medical reporting tool 302 can autofill as much information into the report as available in or determined from existing medical records for the patient (e.g., included in the one more storage databases 303, the patient profile and past Dx/Tx data 324 and/or the usage logs 332). As described in greater detail below, the one or more NLU models 320 and/or the one or more clinical language models 330 can include or correspond to pre-trained LLMs

In some embodiments, using generative AI techniques performed via one or more NLU models 320 and/or the one or more clinical language models 330, the intelligent medical reporting tool 302 can further generate structured, relevant and required content for inclusion in the report/record that ensures compliance with the applicable clinical guidelines and standardized medical ontologies in a concise manner that facilitates efficient and consistent interpretation by disparate healthcare professionals and organizations. For example, given a predefined prompt in the report/record template, such as a prompt indicating a particular category of information to be described in free-text box (e.g., a prompt requesting a summary of the patient encounter, a prompt requesting a summary of patient symptoms, a prompt requesting a recommended treatment plan, etc.), existing medical records for the patient, and applicable clinical guidelines and ontologies for the report type, a pre-trained LLM can automatically generate the corresponding content.

While a user is interacting with the report/record, the intelligent medical reporting tool 302 can continually adapt its suggestions and information in real time. This provides real-time assistance to medical professionals as they are working on creating and updating medical materials. In addition to suggestions and edit recommendations, the intelligent medical reporting tool 302 can provide annotations of information, text, images, and other data objects while preparing the report. This helps to reduce the time to annotate the reports after completing the reports by doctors.

In this regard, in association with automatically integrating and generating relevant information and receiving information entered by the user, the intelligent medical reporting tool 302 can identify and facilitate correcting reporting issues (e.g., via providing feedback and/or automatically correcting certain issues), including errors attributed to omitted information, ambiguous information, extraneous/irrelevant information and incorrect information, including incorrect information attributed to inconsistencies between information within the report itself, the patient's prior medical history, and applicable clinical guidelines and ontologies. For example, using AI technology, the intelligent medical reporting tool 302 can identify and facilitate correcting spelling and grammatical errors as tailored to the applicable clinical ontologies and guidelines. However, as opposed to merely correcting spelling and grammatical errors, the AI assistance provided by the intelligent medical reporting tool 302 can identify and facilitate correcting conceptual errors as tailored based on the type of the report/record, the applicable clinical guidelines and ontologies, and the patient based on existing information known about the patient and the patient's medical history.

For example, the intelligent medical reporting tool 302 can identify inconsistences between information entered by the user within the report, such as initial description of the patient's pathology involving the right lung and later referring to the left lung. The intelligent medical reporting tool 302 can also identify inconsistencies between information entered by the user that does not comply with known information about the patient as included previous reports, existing information in the patient's EMR/EHR, medical imaging records, laboratory records, and the like. The intelligent medical reporting tool 302 can also identify inconsistencies between information entered by the user that does not reflect applicable medical guidelines, such as information providing a medical interpretation or diagnosis of the patient that is inaccurate with respect to the applicable clinical guidelines or ontologies given what is known about the patient. In another example, the intelligent medical reporting tool 302 can identify errors attributed to usage of incorrect medical terminology, such as usage of the wrong medical term based on the definition of the term, as provided in a standardized medical ontology or dictionary, and known information about the patient. Similarly, the intelligent medical reporting tool 302 can identify and facilitate correcting usage of ambiguous terms, acronyms, and replace similar terms with standardized terms in accordance with applicable clinical guidelines and ontologies.

The intelligent medical reporting tool 302 can also identify missing information that is required for the report based on the type of the report and an applicable clinical guideline and suggest or automatically enter the missing information. The missing information can be based on a requirement for the report or contextual based on AI generated content and/or user provided content during generation of the report. In an example, if the user provides information describing a particular event that occurred in the patient's treatment and the applicable clinical guideline for the report requires the event to include a date and location of occurrence, the intelligent medical reporting tool 302 can determine that the date and location information is needed at the time of entry of the event information and automatically enter this information (as retrieved in existing medical records for the patient) and/or prompt the user to add this information. The intelligent medical reporting tool 302 can also identify extraneous or irrelevant information that is included in the report based on the type of the report and an applicable clinical guideline and suggest removing or automatically remove the extraneous/irrelevant information. Using generative AI technology, the intelligent medical reporting tool 302 can also rephrase user entered information in a more concise and accurate manner that is consistent with applicable clinical guidelines and ontologies.

In various embodiments, the intelligent medical reporting tool 302 can provide feedback regarding identified errors and edit recommendations for correcting them during report generation that enables the user to correct the errors. The intelligent medical reporting tool 302 can similarly identify and provide feedback and edit recommendations regarding errors identified in previously generated reports in association with editing previously generated reports. Such feedback may be in the form of a visual indication such as highlighting, underlying or the like, with distinctive indicators (e.g., different colors, icons, patterns, etc.) that reflect different types of errors. For example, the different types of errors can be predefined and include, but are not limited to, an ambiguity error, a vocabulary error, a grammatical error, a spelling error, a contradiction error, an incompleteness error, a wrong information error, an extraneous/irrelevant information error, and the like. In some embodiments, the feedback information can also include auxiliary information describing the error and/or a basis for the error. For example, the auxiliary information can include a description of the basis for the error and in some implementations, include a link or hyperlink to relevant information providing the basis for the error. The relevant information can include a relevant portion of the patient's medical history (e.g., which may include text data, medical image data, laboratory data, and so on) and/or a relevant portion of the applicable clinical guidelines or ontologies. For example, as applied to contradiction errors based on inconsistencies between information included in the report and existing information known about the patient or an applicable clinical guideline or ontology, the feedback information can include a link to the corresponding existing information (e.g., an entry in the patient's EMR, a radiology report, an entry in the radiology report, a medical image showing a particular pathology, and the like) or the portion of the applicable clinical guideline/ontology for which the inconsistency is based. The feedback can also include edit recommendations regarding how to correct the errors which can be accepted, rejected or edited by the user.

The intelligent medical reporting tool 302 can also provide feedback regarding AI generated content integrated into the report, including feedback identifying the AI generated content as well as feedback that indicates a measure of confidence in the accuracy of the AI generated content. Such feedback may be in the form of a visual indication such as highlighting, underlying or the like, with distinctive indicators (e.g., different colors, icons, patterns, etc.) that reflect the content was AI generated and corresponding measures of confidence in the AI generated content (e.g., a red color may be used to indicate a low confidence level, a yellow color may be used to indicate medium confidence level, and a green color may be used to indicate a high confidence level). These types of visual confidence indicators are easier to interpret by medical professionals as opposed to technical confidence metrics used by AI developers (e.g., confidence scores, confidence percentages, Dice scores, etc.). Additionally, or alternatively, such technical confidence metrics can be used. In some implementations, the feedback can also include a link or hyperlink to relevant information providing the basis for which the AI content was generated, such as relevant information included medical records for the patient, including text data, laboratory data, image data, and other types of data), and relevant portions of applicable clinical guidelines and ontologies. In association with reviewing the AI generated content, the user can accept, reject, or edit the AI generated content.

User interface examples provided by the intelligent medical reporting tool 302 for generating and editing medical reports/records are shown in, at least, FIGS. 13-17. These show examples of edit suggestions and recommendations the intelligent medical reporting tool can provide, in various embodiments. The edit suggestions can be based on errors detected in user provided report/record content and/or AI generated report/record content. In the examples shown in FIGS. 13-17, the edit recommendations give a user a concise list of improvement edits. In some embodiments, a user can interact (such as a mouse click, finger touch, right click, finger swipe, or other inputs depending on the computer hardware and software involved) with the suggestions to learn more about why the suggestion was provided. In these circumstances the intelligent medical reporting tool 302 can provide the medical reasoning, ontology, grammar, or other reasoning of why the suggestion was made so a user can have further confidence and understanding of the suggestion.

FIG. 15 shows a comparison between original report/record content 1502 and a marked-up version 1504 of the report/record content 1502 generated by an annotation module 1506, according to an embodiment. The original report/record content 1502 can correspond to user entered content or AI generated content. In some implementations the original report/record content 1502 can correspond to content entered by a user using the intelligent medical reporting tool 302 while creating a report/record for a patient and the marked-up version can be generated by the annotation module 1506 in real-time. In other implementations, the original report/record content 1502 can correspond to content included in a previously generated medical report/record that was stored (e.g., in one or more storage databases 303 and/or patient profile and past Dx/Tx data 324) and accessed by the intelligent medical reporting tool for editing/annotating. In this regard, in some embodiments, the system 300 can include the annotation modules 1506 and/or the intelligent medical reporting tool 302 can include or otherwise access and employ the features and functionalities of the annotation module.

In this example, the marked-up version 1504 includes annotation data (e.g., indicated in boxes with capitalized text) associated with certain terms or phrases in the original report/record content 1502. The annotation data includes or corresponds to labels or tags that provide a corresponding classification of entities (e.g., terms or phrases) that have been recognized by the annotation module 1506 using a named entity recognition process (e.g., performed using one or more NLU models 320 and/or one or more clinical language models 330). These tags or labels can include or correspond to standardized classification tags or labels for the recognized entities as defined in accordance with one or more standardized medial ontologies. In this regard, using a named entity recognition process (e.g., performed using one or more NLU models 320 and/or one or more clinical language models 330), and one or more medical ontologies 322 (e.g., UMLS or the like), the annotation module 1506 can identify and map recognized entities to their corresponding standardized classification labels or tags.

The annotation data does not correspond to detected errors or suggested edits for errors. On the contrary, the annotation data corresponds to classification labels or tags for recognized entities (e.g., terms or phrases), wherein the classification labels are determined in accordance with a standardized medical ontology. These labels or tags can be associated with the corresponding terms or phrases in the original report/record content as metadata or a separate file. These labels or tags can be used by the intelligent medical reporting tool 302 to facilitate detecting errors and/or generating additional content (AI generated content) to be added to the report/record. These labels or tags can also be used by downstream AI algorithms to perform various inferencing tasks with greater accuracy and without requiring pre-processing to perform named entity recognition, thus increasing inferencing speed and reducing costs (e.g., in terms of amount of computational resources used) and time at inferencing.

In some embodiments, the intelligent medical reporting tool 302 can present the original report/record content 1502 and/or the marked-up up version to a user via a graphical user interface of the intelligent medical reporting tool 302. A user can review and interact with the suggested classification labels or tags to accept, reject or edit them. Thus, the user can provide feedback during report/record generation and/or editing regarding the accuracy of the AI generated annotation data, which can be logged (e.g., in usage logs 332) and used to update or tune respective AI models used to perform the named entity recognition process.

Returning to FIG. 3, in association with facilitating generating a new medical report/record by a user and/or editing an existing medical report/record by a user using the intelligent medical reporting tool, the clinical information extraction module 304 can receive text data and structured data from the intelligent medical reporting tool 302. The text data and the structured data can include or correspond to data that has been entered into the report/record by the user and/or AI generated content. The text data and the structured data can also include or correspond to template data that is included with a template used for the report/record, such as predefined data fields and predefined information prompts included in the template. The clinical information extraction module 304 can also receive medical data structures 318 and NLU models 320. The clinical information extraction module 304 may put any received or extracted information to storage data, including in medical data structures 318.

The clinical information extraction module 304 further takes in additional data and information from a myriad of sources (some healthcare related some may not be) to facilitate detecting errors, generating annotation data and creating AI generated content for inclusion in the report/record. This is an important part of the system 300 as it helps an intelligent medical reporting tool 302 be context aware. Information may be provided to and from one or more data storage databases 303, NLU model updation module 316, medical data structures 318, NLU models 320, medical ontologies 322, patient profile and past Dx/Tx data 324, clinical guidelines 326, synoptic reporting standards 328, clinical language models 330, usage logs 332, and user profile and preferences 334. A context aware intelligent medical reporting tool 302 better provides enriched information and edit recommendations to a user.

Other sources of data may include the following. Medical ontologies help recognize medical vocabulary and medical reasoning. If a medical word is identified, the system 300 can standardize the mentions, and provide an on-screen definitions and materials. And, based on the user behavior, the system 300 can learn and become more accurate. Patient profile and past EMR records, images and test results of a patient allow for personalization of edit recommendation to the patient's situation. Clinical guidelines (both global and specific to a hospital or practitioner) help with customization of edit recommendations to standard clinical best-practices. Historical information known about the patient already included in the patient's EMRs (doctor, departmental, hospital specific) and/or other sources help with customization of edit recommendations to personal/department styles and treatment approaches. User profile and usage logs of user activity allow for personalization of edit recommendations to personal styles and preferences. Any current information (e.g., user generated content, AI generated content, and template content) integrated into a medical report/record being generated or edited (even if only partially filled) is used by the intelligent medical reporting tool 302 to customize edit recommendations using one or more LLMs (e.g., NLU models 320 clinical language models 330). For example, language models (with support of enriched text) can identify errors and generate edit recommendations, generate annotation data and automatically generate content to be included in the report/record as needed (e.g., as required based on the template, the applicable clinical guidelines, the existing information, user entered information, etc.) or appropriate. Synoptic reporting standards 328 such as template suggestions can help identify mandatory items.

In various embodiments, the clinical information extraction module 304 can determine and output for a given input (e.g., text string) a statement identification, a subject, predicate, object, source text segment, text start position, confidence score, and consequent statements. The confidence score may be from an AI or ML model that has provided the analysis, and providing a confidence level that the labelling assigned or determined metadata is accurate. Additional label information may be metadata in some embodiments.

FIG. 9 shows further detail in regards to the clinical information extraction module 304, in an embodiment. In this regard, FIG. 9 presents a flow diagram of an example process 900 that can be performed by the clinical information extraction module 304 in association with processing text included in or associated with an EHR 902 or portion thereof. The EHR 902 can correspond to a previously generated record/report for a patient and stored in one or more storage databases 303. Additionally, or alternatively, the EHR 902 can correspond to a new report or medical record being created using the intelligent medical reporting tool 302.

Process 900 involves entity recognition at 904, assertion recognition at 906, relation extraction at 908, FIG. 9 includes EHR 902, entity recognition 904, assertion recognition 906, relation recognition 908, and ontology linking at 910. The clinical information extraction module 304 takes in electronical health record text in the EHR 902. The clinical information extraction module 304 further processes the text in the EHR 902 using one or more pre-trained LLMs (e.g., NLU models 320 and/or clinical language models 330). The one or more pre-trained LLM can be previously trained from large clinical datasets and may be tailored to particular types of reports, clinical datasets, medical concepts (e.g., different pathologies, diagnoses, medical conditions, etc.), patients, medical departments, clinicians or the like. Fore example, the LLMs may be fine-tuned with granular oncology-specific parameters (or other medical expertise), specially annotated and augmented textual data, followed by being mapped to oncology-specific (as an example) dictionaries, taxonomies, and ontologies.

Entity recognition 904 at involves analyzing the text via an LLM (e.g., one or more NLU models 320 and/or clinical language models 330) to recognize entities that may be classified in one of a number of ways. In the example shown, a medical condition (TUMOR) is found in the text, along with a location (POSITION) and part of the body (BODY PART). This then allows the clinical information extraction module 304 to look through other classifications and related databases for TUMOR, POSITION, and BODY PART, in this example. In some embodiments, these entity classifications can be associated with the corresponding terms or phrases in the text annotation data (and integrated therewith as metadata or a separate file). The entity classifications can also be used by the information extraction module 304 to more clearly analyze information and understand it based on its recognized entity type.

Assertion recognition at 906 involves analyzing (e.g., using one or more LLMs) a subset of text that has a recognized entity type classification to determine and assert additional information about the context of the recognized entity. For instance, in an example as applied to recognition of a term corresponding to TUMOR, based on analyzing additional text associated with the entity, the one or more NLU models 320 and/or clinical language models 330) may assert that the TUMOR is PRESENT.

Relation recognition at 908 involves determining relationships between different parts of the text that have been recognized as entities. For example, relationships of time, location, to which patient, by which doctor, which specialty of medicine, which drugs and dosages, and other relationships can be determined by the clinical information extraction module 304 via relation recognition at 908. In the example of FIG. 9, the output of the relation recognition at 908 includes a determination that the TUMOR has been found AT the POSTION and INTO the BODY PART. Thus, a nodular tumor extension (TUMOR) has been found AT 8 to 10 o'clock position (POSITION) and INTO the mesorectat fat (BODY PART). The relationship between data and classification types is very helpful in providing the edit recommendation module with useful context.

Ontology linking at 910 involves further connecting or mapping certain recognized entity types to other clinical concepts that are related to the entities in medical dictionaries, taxonomies and/or ontologies. The related concepts can include medical codes, proper medical names, or other agreed upon terminology or verbiage. In the example of FIG. 9, the tumor extension is linked with a standard SNOWMED CT code along with a reference number 38541300, and the position is linked with a standardized radiology identifier for the position, that is RADLEX RID6028. SNOMED CT is one of a suite of designated standards for use in U.S. Federal Government systems for the electronic exchange of clinical health information. RADLEX relates to radiology lexicon. Radiology Lexicon (RADLEX) contains individual terms and concepts related to medical imaging that can be used in a variety of applications, including radiology reporting, data mining and research. RADLEX Playbook combines RADLEX terms, using a consistent syntax, to provide a standard set of exam names and codes. Through the exemplary steps of process 900 illustrating one embodiment, the intelligent medical reporting tool 302 can gain context about the information coming into the system 300 and build some groundwork understanding that is needed to facilitate auto-generating content to be included in a medical report/record, identifying errors or potential errors, and determining edit recommendations for the errors and annotation support.

Returning to FIG. 3, the enriched information overlay module 306 takes in text data, structured data, and medical data structures 318. Further, enriched information overlay module may take in the output of other modules, such as clinical information extraction module 304. Enriched information overlay module 306 take the input data and finds additional relevant information within the system 300 or from outside the system (e.g., medical dictionaries on the internet, medical guidelines in the hospital intranet, or other external systems). Then, the enriched information overlay module 306 can provide additional labels and a deeper level information (if requested by the user)—enriched information—about the content of a medical report. Such labels (also included in enriched information) can be entity labelling for words and phrases, section labelling, identifying the relationship or association between entities, identifying negations, weak assertions, hypotheticals and conditionals, identifying temporal assertions, as well as identifying the ontological concept identification for entities. This enriched information can be provided to intelligent medical reporting tool 302 and displayed to users for interaction. FIG. 14 shows an example of how the enriched information overlay module 306 can positively impact the medical report/record generation and/or editing processes provided by the intelligent medical reporting tool 302.

FIG. 14 shows details of the enriched information overlay module 306, according to an embodiment. FIG. 14 includes enriched overlay output data 1400, imaging test label 1402, entity label 1404, assertion 1406, temporal assertion 1408, mapping to medical ontologies 1410, section header label 1412, body part label 1420, body part label 1422, body part label 1424, body party label 1426, imaging findings label 1430, imaging findings with body probability pointer 1432, laterally label 1436, structure of reference label with body direction label 1440, structure of reference label 1442, and structure of reference label 1444. Imaging test label 1402 is associated with the CT examination text of the medical report shown in FIG. 14.

Imaging test label 1402 is an example of enriched information overlayed on report text on a display screen. Imaging test label 1402 includes an entity label 1404 (Imaging Test), assertion 1406 (n=none), temporal assertion 1408 (←). Thus, imaging test label 1402 is showing the text broken down into categories that can be linked and understood with other information. This is also helpful for clicking on each to get more information. As mentioned above, the enriched information overlay module 306 can seek out and/or receive additional information from internal and worldwide sources that can be provided to the user as more information if labels are interacted with. Imaging test label 1402 is indicating that no previous imaging test of a particular CT type with the corresponding type label CT0498 has been performed for the patient done. Current Procedural Terminology (CPT) codes offer doctors and health care professionals a uniform language for coding medical services and procedures. A user can be presented with the overlay output data 140 and click, in an embodiment, on the CPT code to get listings of other CPT codes and/or more information on the details of this code. This information could be provided the enriched information overlay module 306.

Section labeling may be done by the clinical information extraction module 304 throughout a data record, medical information, medical records, and medical reports. For example, the section of the medical report being evaluated in FIG. 14 is the FINDINGS section. Thus, in this example, the clinical information extraction module 304 has identified the word FINDINGS as a section header and the enriched information overlay module has displayed a corresponding section header label 1412. Various body parts are also labeled as they come in the report, such as body part label 1420, body part label 1422, body part label 1424, and body party label 1426. As body parts are found, more information may be generated based on how many times the body part appears, the vitalness of the body part, relations of the body part to other listed body parts in the report, and other factors to give additional context to the report and help provide better edit recommendations. Imaging findings label 1430 and imaging findings with body probability pointer 1432 both are findings from an imaging scan by a medical device. The body probability pointer 1432 further links the imaging finding label 1430 to a probable body part associated with the imaging findings. This helps the system connect different medical tests to other areas of context for the test, such as the results and/or related body parts. The laterally label 1436 provides information on location and direction of a structure or body party in the report. The structure of reference label with body direction label 1440, structure of reference label 1442, and structure of reference label 1444 give additional information as to where and what part of a body part is being referenced in the report. For example, the structure of reference with body part direction label 1440 indicates that it is the lower part of the rectum, which has body part label 1424. The relationship and/or association between labeled entities can be very important when performing ML or AI analysis as part of an intelligent medical reporting tool 302. These provide additional enriched information both to the user and the intelligent medical reporting tool 302. Enriched information generated by the enriched information overlay module 306 can be provided both to the user as well as to other modules within the intelligent medical reporting tool 302.

Returning to FIG. 3, the edit recommendation module 308 is an important part of an intelligent medical reporting tool 302. The edit recommendation module 308 facilitate detecting errors or potential errors in a medical report/record and determines and provides edit recommendations for the errors to for displaying on a display screen and for a user to interact with using computer input devices. Examples of errors and corresponding edit recommendations are detailed throughout the specification and drawings, including with reference to FIGS. 13-17. The edit recommendation module 308 takes in information from throughout the system, such as from contradiction and incompleteness detection module 312, language model finetuning module 314, and user profile and preferences 334.

The edit recommendation module 308 can provide edit recommendations in real-time as a user as the user is generating and/or editing medical material, such as a medical report or medical record. Thus, the edit recommendation module 308 is an element that helps the intelligent medical reporting tool 302 suggest edits and labels/annotations while a healthcare professional is writing or reviewing a report/record. One example of edit recommendations is related to the FHIR standard. The FHIR (Fast Healthcare Interoperability Resources) standard defines how healthcare information can be exchanged between different computer systems regardless of how it is stored in those systems. In some embodiments, If the clinical information extraction module 304 identifies certain parts of the materials may relate to FHIR or certain fields of a report, the edit recommendation module can auto-fill or auto-populate FHIR concepts. The codified FHIR concepts can be parsed from the report and the edit recommendation module 308 can auto-fill FHIR data of reports, which is a much needed feature both for productivity of doctors as well as enormous benefit to downstream applications.

FIG. 10 shows details of the edit recommendation module 308, according to an embodiment. In various embodiments, the edit recommendations module 308 includes word/sentence/template completion module 1002, template detection module 1004, sentence rephasing module 1006, ambiguity reduction module 1008, contradiction elimination module 1010, completeness ensuring module 1012, and edit explanation module 1014. The edit recommendation module has many types, or categories, of edit recommendations. These may be related to many types of issues discussed herein throughout, including, but not limited to, ambiguity issues, contradiction issues, incompleteness issues, rephasing issues, and auto completion of words, template-based recommendations, thoughts, phrases, paragraphs, or sentences.

The word/sentence/template completion module 1002 is enabled by one or more LLMs (e.g. one or more NLU models and/or one or more clinical language models 330), neural networks, AI, generative AI, rules engines, and/or machine learning. This may be through language model finetuning module 314 in an embodiment. LLMs have acquired an embodied knowledge about syntax, semantics and ontology inherent in human language. LLMs that help identify edit-candidates may be masked (MLM) or causal LLMs. Causal language modeling is the task of predicting the token following a sequence of tokens.

Further, based on previous histories of reports for the patient or similar patients and (e.g., as included in one or more storage databases, patient profile and past Dx/Tx data 324) the type of the report, applicable clinical guidelines and ontologies, and information included in usage logs 332 of a user or department, patterns can be recognized and/or clinical guidelines may be followed. The word/sentence/template completion module 1002 can suggest text and other outputs that may be what the user would prefer and/or intend for the rest of the word, sentence, or template to have in it. This may provide helpful auto-complete suggestions for partially typed words, partially typed sentences, partially typed templates, and various report fields defined in the report. The type of report and related template may be detected by the system through the template detection module 1004. In certain medical environments there may be a library of relevant templates that template detection module 1004 can be aware of and help to pre-populate with auto-generated AI content as extracted from various sources (e.g., past medical records, clinical guidelines and ontologies, information included in usage logs, etc.)

The sentence rephasing module 1006 analyzes sentences and can suggest rephasing improvements. This may be based on best practices learned from analyzing other reports, based on proper medical phrasing, or based on language issues. Exemplary issues that the sentence rephrasing module 1006 addresses include a lack of conciseness, lack of clarity, reformatting changes, the need to align to a different template or format, incorrect use of proper medical terminology, and others. Further, if the language of the report is not the first language of the user, sentence rephrasing module 1006 can provide helpful suggestions for the language of the report.

The ambiguity reduction module 1008 analyzes the report/record and provides suggestions to help improve clarity and reduce ambiguity. The ambiguity reduction module 1008 may address problems such as the presence of an unknown word, incomplete or ambiguous sentences, pronoun issues, anaphora, ellipsis, relative temporal references, polysemous word, acronyms, and abbreviations. Detection and remediation of such problems may be in sub-modules or sub-routines. FIG. 10 shows ambiguity reduction module 1008 including sub-modules of unknown word handler, polysemy handler, and anaphora handler.

The contradiction elimination module 1010 analyzes the report/record and provides suggestions for edit recommendation module 1000 to remove contradictions. The contradictions may include statements contradicting with other parts of the report, statements contradicting with existing patient information, statements contradicting with medical ontologies, statements contradicting with clinical guidelines, formatting inconsistencies within the document, formatting contradictions with synoptic standards. For example, if the patient is male and the report says the patient is female, or report says the patient has ovarian cancer, then there may be a contradiction. Sometimes a contradiction may be the result of an error, like a mistype. For example, one part of the report may say a patient weight is 176 pounds and another part of the report may say the patient weight is 17 pounds. The contradiction elimination module 1010 may notice the contradiction and also suggest an edit for display on the screen that the second weight (17) should be updated, potentially to 176.

The completeness ensuring module 1012 analyzes the report/record and provides suggestions for the edit recommendation module 1000 to assist with completeness. Completeness can be determined based on the type of the report and any requirements for the type of the report or associated content included therein as controlled based on a template for the corresponding report and/or one or more applicable clinical guidelines. Incomplete reports/records can cause medical issues and waste time. Incompleteness issues solved by the completeness ensuring module 1012 may include a lack of cross-reference links (e.g., to prior reports, images or medical reference data sources), incompleteness in complaint, diagnosis or treatment history, broader concept in place of more a granular concept, and missing aspects compared to reporting standards. Many medical professionals are under a time crunch and may be tempted to not fully complete orders, reports, and records. By providing specific edits and suggested text, they may feel less burden as they work to complete the word, thus improving patient care and speeding up medical workflows. One type of suggestion that may be output on a user screen is to automatically follow up with another hospital or physician. In some embodiments, based on detection of missing information (and thus incompleteness) in a medical report or record, the system 300 can automatically generate a communication to the appropriate entity or entities likely to have the needed information (e.g., appropriate clinical IT systems, medical provides, clinicians, patient's themselves, etc.). Then an edit recommendation can be shown on the screen regarding the missing information. For example, if information is needed from a particular medical imaging scan performed for a patient but the imaging data is not available to the system 300, the edit recommendation may include a message regarding the need for obtaining the scan data, such as “Contact Dr. Faber for a copy of the imaging scan”. In some implementations, the message can be interactive and facilitate performing the requisite action. For example, if the user clicks on the recommendation message, a draft email or other communication type can be shown on screen already filled out and ready to be edited and/or sent. This can help with completeness of records without burdening medical professionals. Completeness ensuring module 1012 may include the sub-modules of concept elaboration (adding more text when a concept is not fully elaborated), cross reference linker (if multiple sections of a report can be cross-referenced for better understanding a link is provided), and conformance enforcing.

The edit explanation module 1014 provides additional information and context to why a certain edit was recommended. A user using the intelligent medical reporting tool may not understand why a suggestion was recommended or how the recommendation was generated. The edit explanation module 1014 can generate links or hyperlinks to relevant information for which edit recommendations are based and associate the links or hyperlinks with the corresponding edit recommendations. By clicking on the edit recommendation and/or associated link (or provided automatically without a user input prompt), the intelligent medical reporting tool 302 can provide (e.g., render in a pop-up display, a new window, or the like) the explanation information provided by the edit explanation module 1014. Such explanations may explain why a cross reference was made and may provide a link to the relevant patient data model and clinical guidelines. The modules shown in FIG. 10 are not an exhaustive listing, and other features herein throughout may be provided by edit recommendation module 308.

Returning again to FIG. 3, the usage and feedback logging module 310 is vital for processing the requests of the user, as well as training the AI models employed by the system (e.g., one or more NLU models 322 and/or clinical language models 330) for future edit suggestions. How each user interacts with the system in terms of accepting, rejecting and/or editing AI generated content and/or edit recommendations is received from the intelligent medical reporting tool 302 as user decisions and user feedback. The user feedback can also account for learned information from the user regarding what is preferred for inclusion in different types of reports, preferred writing styles, preferred terminology and the like. The usage and feedback logging module 310 records the user decisions in usage logs 332 and/or user profile and preferences 334. Both the explicit requests from the user may be logged, as well as the individual usage pattern (clicks, touches, visual contact, and any other type of input) of the user. In some embodiments, the usage and feedback logging module 310 can provide a feedback request requesting feedback from the user (e.g., via a pop-up windows or other on-screen or voice feedback mechanisms) during report generation and/or editing (or upon completion) asking the user to provide feedback regarding how the report generating/editing experience was and/or requesting improvement suggestions on specific edit recommendations. These are especially valuable when a user does not accept recommended edit and/or AI generated content. In some implementations, feedback requests integrated with respective edit suggestions, AI generated content and/or the report generation/editing session as a whole, may have different types of feedback questions that can facilitate guiding the user responses, such as “Is there is an error?, “Is there something missing?”, or “Was the AI generated data provided granular enough, or too granular?” and the like.

The contradiction and incompleteness module 312 may be its own module as shown in FIG. 3 in an embodiment, or may be a part of the edit recommendation module 308, as shown in FIG. 10, in an embodiment. In this regard, the contradiction and incompleteness module 312 can provide the same or similar features and functionalities as the contraction elimination module 1010 and/or the completeness ensuring module 1012 as described above. For example, the contradiction and incompleteness module 312 can detect discrepancies between the information that is included in the report/record (e.g., entered by the user during report generation and/or as included in a previously generated report/record being reviewed and edited) and the information known about the patient (e.g., as available in historical records for the patient), applicable clinical guidelines and medical ontologies.

FIG. 17 shows an example graphical user interface 1700 of the intelligent medical reporting tool 302 in accordance with an example embodiment. In this example, the intelligent medical reporting tool 302 is being used in association with generating a new report for a patient by a user. In this example, the report template includes a free-text box 1702 within which the user can enter relevant text for the report, which may be a description of a patient encounter, clinical notes, or the like. The free-text box in this example is associated with a predefined subject field which corresponds to “INTRODUCTION.” In this example, the user has begun generating the report by typing within the free-text box 1702 with the introduction predicate.

In this example, the contradiction and incompleteness module 312 has identified a contradiction 1704 with the introduction of the medical report 1702. Based on detection of the contradiction, the intelligent medical reporting tool (or more the contradiction and completeness module 312) has generated and displayed a visual icon (i.e., the X mark) indicating the detected contradiction. A pop-up window 1706 or side-bar window may be displayed to explain the detected contradiction or error. In some implementations, an edit suggestion may be determined and presented to the user for correction. For instance, in this example, the contradiction is based on the patient being male and thus not medically possible to have ovarian cancer. Thus, for instance, the contradiction and incompleteness module 312 can determine and suggest changing the word “male” to “female” in the report. For example, the graphical user interface 1700 includes various toggle switches, including a regime suggestor switch. The X mark indicating the contradiction is further rendered next to the regime suggestor switch. In this example, the user can select the regime suggestor switch to review suggested corrections to the contradiction 1704

The toggle switches 1708 also include a domain knowledge switch and an enable auto annotations switch. also shows toggle switches related to features of an intelligent medical editor. The toggle switches enable the user to control the edit recommendations and can tweak features of the intelligent medical reporting tool 302 based on their preference. For example, the domain knowledge switch can be used to activate or deactivate integration and usage of domain knowledge (e.g., clinical guidelines, ontologies, and other types of domain knowledge) in association with auto-generating content and detecting errors and providing edit recommendations. The enable auto annotation switch can be used to activate or deactivate integrating annotation data with recognized entities in the report (e.g., as metadata or a separate file).

FIG. 11 shows an example of the contradiction and incompleteness detection module 308, according to an embodiment. As noted above, the contradiction and incompleteness detection module 308 can perform factual contradiction and incompleteness detection using existing medical records for respective patients, applicable clinical guidelines and medical ontologies. In addition to the corresponding discussion in FIG. 3 and FIG. 10, FIG. 11 gives additional technical detail related to the contradiction and incompleteness detection module 308. As shown in FIG. 11, in one or more embodiments, the contradiction and incompleteness detection module 308 can include knowledge graph comparison module 1102, knowledge graph creation module 1104, guideline conformance detection module 1106 (further discussed in relation to FIG. 12), standardization of formatting module 1108, standards incompatibility detection module 1110, standardization of mentions of medical terms module 1112, cross-reference finding module 1114.

The knowledge graph comparison module 1102 may take in the output of clinical information extraction module 304 and include many fields, such as statement ID, subject, predicate, object, source text segment, text start position, confidence score, and consequent statements. A knowledge graph in the system 300 may be compared with a knowledge graph for the specific record being edited to find contradictions and incompleteness. Such knowledge graphs may be created by knowledge graph creation module 1104, which may take in data about the patient (such as structured data including statement ID, date, subject, predicate, object pointers to the source information, patient journey stage, confidence score, and consequent statements). A knowledge graph specific to a patient may also be thought of as a patient data model, in one embodiment. Knowledge graphs may be generated and compared using semantic networks. A semantic network is a knowledge structure that depicts how concepts are related to one another and illustrates how they interconnect. Semantic networks use artificial intelligence (AI) programming to mine data, connect concepts and call attention to relationships. This can include tagging parts of speech in the text (in implementations where speech to text is used for data input); recognizing known words in the text; and creating a semantic network, the semantic network including at least one of the recognized known words and at least one relationship with at least one semantic concept associated with at least one of the recognized known words; and wherein the output pertains to a probability of a particular occurrence.

The incompleteness detection and report structure recommendations may be provided by standardization of formatting module 1108 using synoptic reporting standards, such as synoptic reporting standards 328. Standardization of mentions of medical terms module 1112 reviews the usage of medical terms and provides suggested edits to align the common usage, spelling, and abbreviation of medical terms in the report. Standards incompatibility detection module 1110 compares the report with medical industry standards, such as billing codes, cariology guidelines, medical rules, and hospital requirements. Cross-reference finding module 1114 detects if there should be a cross reference between information items and provides an edit suggestion to add the cross reference.

Returning to FIG. 3, language model finetuning module 314 works with clinical language models 330, industry data/standards, and usage logs 332 to update and fine tune the respective clinical language models 330 over time. As noted above, the one or more clinical language models 330 can include LLMs with generative AI capabilities to continually update and fine tune the respective language models. Language model finetuning module 314 provides the edit recommendation module 308 with specific language to use in edit recommendations based on the type of report, the user involved, context of the text, and other factors such as medical ontologies 332, medical data structures 318, patient profile and past Dx/Tx data 324, clinical guidelines 326, synoptic reporting standards 328, and user profile and preferences 334. The language model finetuning module 314 allows the intelligent medical reporting tool 302 to re-align reports to different templates and writing or report styles using LLMs. For example, in some embodiments, different report templates and writing or report styles can be defined and/or learned. The different report templates can account for variations of a same type of report used across different medical providers, medical departments, organizations, hospital systems, and the like. The different writing or reporting styles can also account for these factors, as well as more granular writing styles tailored to individual users. In some embodiments, the language model fine tuning module 314 can generate different versions of LLMs configured to apply the different report templates and/or writing styles. The language fine tuning module 314 can also generate global LLMs that aggregate different report templates and writing styles learned for different organizations, medical providers, users, etc., as tailored to a particular type of report. The language model finetuning module 314 may be connected with and share bi-directional information with NLU model updation module 316 as both modules deal with natural language.

Medical data structures 318 may be of many types depending on the medical EMR company, IT suppliers, government regulations around medical data, what fields may be needed for certain disease types, varying report record types depending on medical specialty, database requirements, and other factors.

The NLU model updation module 316 receives usage logs 332 as well as Federated NLU models. NLU model updation module 316 provides updated NLU models to NLU models 320. These NLU (natural language understanding) models may include NLP (natural language processing), NLG (natural language generation) and NWP (natural word processing) models as well. These will be further explained in connection with FIGS. 5, 6, 8A, 8B, and herein throughout. The intelligent medical reporting tool 302 employs one or more NLU models 320 in to perform natural language processing, content generation, and understanding. For example, in various embodiments, the NLU models 320 are used by the clinical information extraction module 304, the enriched information overlay module 306, the edit recommendation module 308, usage and feedback logging module 310, the contradiction and incompleteness detection module 312, and language model finetuning module 314 to perform the operations described with respect to the corresponding modules.

In some embodiments, the patient profile and past Dx/Tx data 324 corresponds to a collection of all the patient data available for patients throughout the system (e.g., historical EMR/EHR data for the respective patients, historical imaging data, historical laboratory data, and so on). Additionally, or alternatively, such historical information for respective patients may be stored and access in one or more storage databases 303. The more accurate and comprehensive historical data is for a given patient, the more insights and recommendations the intelligent medical reporting tool 302 can provide in association with generating new reports for the patient. Patient profile data may be from an EMR, medical reports, medical imaging scans, medical notes, communication from a patient, and more. Patient profile data can include their allergies, problems, medical history, medications, and various other types of clinical and non-clinical information. FIG. 19 discusses more about patient data present in hospital premises and using patient profile data to determine insights and recommendations.

User profile and preferences 334 stores and analyzes information about past, current, and future users of the intelligent medical reporting tool 302. Each user may have a different role (such as nurse, primary care physician, radiologist), a different aptitude for using software technologies, a different knowledge of medical terminology, a different knowledge of medical standards, and the like. User profile and preferences 334 takes information in from usage and feedback logging module 310 and can provide output to any module, but in specific to edit recommendation module 308. An edit related to a report may be different based on the user involved at times. For example, one user consistently uses abbreviations vs. another user always types out the full word. Or if one user has a pattern of using a certain medication to treat the same patient disease, it can be suggested as a recommendation in the future. Thus, pattern recognition and artificial intelligence learning is helpful for user profile and preferences 334, and the module may have such technology built in.

FIG. 4 shows a flow diagram of an example process 400 performed by the intelligent medical reporting tool 302, according to an embodiment. Process 400 involves a user 404, clinical incorrectness detection 406, clinical warnings 408, natural language understanding 410, natural language generation and natural word processing (NWP)—NLG 412, medical ontologies 414, causal pair 416, patient history 418, clinical care guidelines 420, and vocabulary (vocab) standardization 422. 555

The user interacts 404 interacts with the intelligent medical reporting tool 402 using any suitable computing device that includes input/output capabilities and network communication capabilities (e.g., computer 2602 shown in FIG. 26 or the like). The user 404 can use the intelligent medical reporting tool 402 to generate, edit, and improve medical materials, such as radiology reports, electronic medical records, clinical notes, prescription orders, and others. The clinical incorrectness detection 406 and clinical warnings 408 may be modules that receive information included in the medical report/record being generated or edited as well as from contextual sources such as medical ontologies 414, causal pair 416, patient history 418, clinical care guidelines 420 and other relevant data sources. In the embodiment of FIG. 4, the various modules discussed above with reference to FIG. 3, FIG. 10, FIG. 11 and other figures herein can be integrated with the intelligent medical reporting tool 402. The intelligent medical reporting tool 302 can receive warnings and incorrectness detection and determine edit suggestions with the help of language interaction enabled in the system 300. The intelligent medical reporting tool 302 provides displayed edit suggestions to a user for the user to select and interact with, as shown in FIG. 13, for example.

Clinical incorrectness detection 406 and clinical warnings 408 may have analysis codes with software algorithms and/or AI models in various embodiments. Clinical incorrectness detection 406 leverages medical ontologies 414 and causal pair 416 to find patterns and relationships between information in patient history 418 and medical conditions that are being evaluated in a patient. Clinical incorrectness detection 406 then detects if there is a clinical incorrectness of the information currently in a report and what is medically possible, based on the causal pair 416 and patient history 418. Patient history 418 may receive information signals about a patient in image and/or text forms. An example of a detection of clinical incorrectness is if a patient has no left leg and an imaging report says that the left knee is broken. Causal pairs 416 would detect that the body part was removed during a past surgery (cause), and therefore the current report can't have findings based on that body part. In another example, causal pairs 416 would detect if a patient has ovarian cysts but the patient does not have ovaries. Finding such incorrectness can save lives, time, and the reputation of the medical professional.

Clinical warnings 408 can provide detected warnings to the intelligent medical reporting tool 302 for displaying on a user display. Such warnings may also provide suggested edits and/or changes, in various embodiments. Warnings can be popped up if a treatment decision is taken which is not present in any of the treatment options suggested by applicable clinical care guidelines tailored to the type of the report and the patient's condition and treatment journey. For example, as applied to cancer patients, such guidelines may include the National Comprehensive Cancer Network (NCCN) Guidelines for cancer. The possible treatment options can be found by traversing through the guidelines using the patient history.

Language interaction is enabled through NLG 412, NLU 410, and vocab standardization 422. These are further discussed, among other places, with respect to FIGS. 3, 5, 6, 8A, and 8B. NLG 412 generates natural language and includes a next word prediction module to suggest next words while writing reports. NLG 412 may perform the acts of content determination, document structuring, aggregation, lexical choice, referring expression generation, and/or realization. NLU 310 takes in the input report and patient history and generates a natural language understanding of the content. NLU includes rearranging unstructured data so that the module may “understand” and analyse the data. This natural language understanding thus can be compared against medical ontologies 414 and clinical care guidelines 420 as examples. Medical ontologies may include tables such preferred name (such as Cisplatin/Epirubicin), synonyms (such as Cisplatin/Epirubicin, Cisplatin-e therapeutic implant, CDDP-e TI, and CDDP/EPI), IDs of the medical ontology (such as a web address of a medical body or organization and a link to the specific page), and code (such as C11555). Vocab standardization can take in the medical ontology information, including the synonyms. All the mentions of a medical phrase including short forms or synonyms will be mapped to the standard index from ontologies. For example: non-small cell lung cancer ˜NSCLC˜C2926. This helps in standardizing reports irrespective of writing styles. Further if a user editing a report calls the same thing three different ways (such as through abbreviations or shorthand or slang), vocab standardization 412 can suggest consistent naming throughout a record or report related to the thing.

FIG. 5 shows details regarding the NLU model updation module 316 according to an embodiment. FIG. 5 illustrates a base NLP model 502 (e.g., corresponding to one or more NLU models 320 and/or clinical language models 330), a plurality of remote sites 504, a plurality of model finetuning processes 506, new models 508, and model aggregation 510. The remote sites may correspond to different healthcare providers, organization, departments, or the like. It should be appreciated however that the remote sites 504 can represent individual entities (e.g., individual users) or other group entities (e.g., a group of users of the same medical specialty or the like). The base NLP model 502 is an NLP model that is trained in generic enough tasks to do downstream applications. A goal of the NLU model updation module 316 is to provide the benefits of long-term improvement of NLP due to higher volume of manually annotated data. This also may help improve downstream clinical applications.

After a base NLP model 502 is provided to or otherwise made accessible for usage by medical remote sites 504, medical staff at the remote sites use the model in their instances of the intelligent medical reporting tool 302. Through the process of accepting, rejecting or revising edit suggestions, corrections, reacting to information, and receiving feedback, the user interactions and feedback is logged and aggregated over time. In some embodiments, the NLU model updation module 316 can further fine tune or update respective versions of the base NLP model 502 as tailored to the respective remote sites 504. This fine tuning can be performed locally at the remote sites 504 or by the system 300. In some embodiments, the respective updated versions of the NLP model 502 can be further aggregated by the NLU model updation module 316. For example, using model aggregation 510 technology, the base NLP model 502 can be improved by aggregating the updates made to the respective versions of the NLP model tailored to the respective remote sites to generate a global NLP model.

In this regard, FIG. 5 shows how NLP models may be improved through crowdsourcing of feedback and improvements. This allows for getting clinicians (oncologists, nurses, and primary care physicians) to participate in crowdsourcing of technology. Ensuring that the AI and NLP models are performing adequately to meet clinical standards is crucial before integrating them with a critical clinical system like EMR or other medical reports. High volume, variety and veracity of manual feedback is essential for validating and continuously improving the AI and NLP models. The two roadblocks which hinder such a collection of feedback are the high time and cost of high-quality manual feedback, and the data privacy requirements which result in administrative and data anonymization overheads. The systems herein provide (1) creating a low-intrusion way for clinicians to provide in-place feedback to AI and NLP as well as by creating a federated method to improve the AI and NLP models. This may eliminate the need to share patient data outside of the organization/legal boundaries, thereby enabling wide crowdsourcing of training/validation data for such models. Thus, improvements may be done in a federated way without compromising the patient privacy. These benefits and features are also relevant to the systems shown in FIGS. 8A and 8B.

That said, each remote site 504 still may have a finetuned model particular to its site, its guidelines, its people, jurisdictional cultural or legal requirements, and the relevant tastes of that site. The system can customize the usage of an intelligent medical editor to the taste of specific users, specific departments, and specific locations.

FIG. 6 shows a diagram of example subsystem 600 of system 300 in accordance with one or more embodiments. System 600 incudes intelligent medical reporting tool 302, user behavior input 608, an NLU module 604, a natural word prediction (NWP) module 602, and downstream tasks 610. The downstream tasks 610 may be many in a medical environment, and include the use of such outputs the intelligent medical reporting tool 302. Downstream tasks may include summarization of previous reports as well as highlighting important information.

The NWP module 602 has the task of predicting an upcoming word(s) (or providing multiple suggestions) based on the input string of text, images, or other input content. The predicted future words are provided to downstream tasks 610. In the simplified example of FIG. 6, NWP module 602 takes in the input of “The boy sat on”, and predicts the next words as possibly “a chair”, “a sofa”, or “a bench”.

The intelligent medical reporting tool 302 may provide input for subsystem 600 from the content of the report/medical record being generated or edited. In addition, a current user behavior 608 provides real time input for subsystem 600. User behavior 608 may influence the NLU module 604 and the NWP module 602 because each user may have a different language style. Because of the varying styles, which may be based on different user roles or specialties, skill sets, education levels and adherence to proper terminology, subsystem 600 may be able to incorporate the styles of users and provide that in the edit recommendations to other users. For example, if there is head user of a radiology department who does their work excellently or wants consistent work across their employees, the system may use NLU module 604 to understand the user behavior 608 of the head user and then provide the staff members of that department related edit recommendations in their medical materials. Further, if government officials or regulators expect medical reports and materials to be written in certain standard ways, this also can be fed into NLU module 604 and NWP 602 to improve edit recommendations for users in accordance with the standard ways. In an embodiment, when edit recommendations come from certain sources or based on certain inputs, they can be color coded or marked in certain ways. For example, if an edit recommendation has three options, a user-customized edit recommendation may be in one color or marking, a head of department edit recommendation may be in a second color or marking, and a government edit recommendation may be in a third color or marking. Providing users a bit of context and information as to the origin of certain edit recommendations by the intelligent medical reporting tool can be helpful their adoption and proficient use of such a tool. It may also help shape behavior.

FIG. 7 shows a block diagram supporting natural report generation, according to an embodiment. FIG. 7 includes system 700 which can correspond to a subsystem of system 300. System 700 includes standard report generation 702, medical information 704, patient EHR DB (database) 706, disease DB 708, text parser 710, language system 712, generative artificial intelligence & large language model service 714 (GAI LLM service), word prediction 716, backend apps (applications) 718, medical dictionary 720, user interface 722, security layer 724, patient history 726, record from other hospital 728, family history 730, and patient physical activities 732.

In various embodiments, the intelligent medical reporting tool 302 may use report generation capabilities of a system like system 700 which includes standard report generation 702. Standard reporting can be done for different types of diagnosis and treatments and clinical concepts. Standard report generation 702 may take in information from text parser that has come to text parser from a variety of sources, such as patient history 726, user input through user interface 722, backend apps 718, and medical information 704. Standard report generation takes report templates or previous report examples and generates a new report. Then it prepopulates areas that it knows are related (such as patient names and other factual data from the record). In association with standard report generations 702, the intelligent medical reporting tool 302 can auto-generate content, detect errors and generate edit recommendations as describe throughout the disclosure. For example, the intelligent medical reporting tool 302 can provide edit recommendations and suggestions for other parts of the report that are not fully known or may need a medical professional to make the selection. Thus, a user can interact with a report that has a large portion of it already completed as auto-filled or auto-generated using LLMs and the techniques described herein. For example, the standard report generation 706 generate the reports with prepopulated information and edit recommendations based on previous user reports, other reports, medical standards, and third-party reports. This allows for the leveraging of best practices. Further, a large amount of time can be saved through such a system. The text parser 710 may use GAI and LLM service 714 as well as word prediction 716 to better provide standard report generation 702 with factual information for prepopulating as well as information to improve edit recommendations. The text parser 710 has direct access to medical dictionaries 720 to help it understand obscure or uncommon medical terminology.

Patient history 726 may include internal and external information. Internal information may include patient EHR records from patient EHR DB 706. Patient history 726 may also include records from other hospital 728, family history 730, and patient physical activities 732. These may be from other devices such as smartphones, smartwatches, activity trackers, medical devices at home or in a medical environment, family history in EHR records of family members, and other locations.

Backend apps 718 may be specific software applications to the disease, country, hospital, care pathway, or other targeted group. Backend apps 718 may be provided by specific vendors and may help with information related to specific reports, such as government required reports that certain vendors have expertise in generating and/or pre-populating. This allows for more efficient and accurate report generation. Further backend apps 718 may be apps related to natural language processing or interaction to help text parser to an improved job handling certain types of information. Security authorization layer 724 allows for proper security and authorization to not allow important user and patient information to leak.

FIG. 8A shows a local model for language fine tuning, according to an embodiment. FIG. 8A includes finetuning system 800 which can correspond to a subsystem of system 300. Finetuning system 800 includes language fine tuning module 802, edit option finding module 804, clinical language models 806, hospital-1 810, radiology reports 812, pathology reports 814, consult notes 816, hospital specific model 820, and hospital specific recommendations 822. Hospitals may have specific recommendations 822 provided to its users by edit option finding modules 804, also called edit recommendation modules, of intelligent medical editors. Hospital specific recommendations 822 may be from hospital guidelines or rules. These recommendations may also be based on detected best practices from medical materials of hospital-1 810 such as radiology reports 812, pathology reports 814, and/or consult notes 816. Thus, hospital specific models 820 can be generated for language and edit recommendations. Clinical models 806 can provide publicly available language models trained on large corpus of medical documents which can then be specialized by language model finetuning module 802 and/or edit option finding module 804 to perform hospital specific downstream tasks.

In various embodiments, the language model finetuning module 802 receives clinical language models 806 along with medical materials specific to hospital-1 810, such as radiology reports 812, pathology reports 814, and/or consult notes 816. Using this information and the clinical language models 806 (e.g., corresponding to LLMs) the language model finetuning module 802 can finetune its language model with hospital specific reports allowing for the adoption of hospital specific recommendations. For example, hospital-1 may be in a location where certain terminology or language is used that is different than other hospitals, and certain vernacular is more common. Another example is if a certain hospital requires a specific radiology report section in its reports that is not required by other hospitals (for insurance or government billing in an example). As another example, models may need to be finetuned to suit for specific application to achieve required outcome (such as clinical word classification, relation assertion, word/sentence classification, etcetera.). The language model finetuning module 802 can provide a hospital specific model to edit option finding module 804, allowing edit option finding module 804 to suggest edits that are best suited to the hospital in question. This may also be done at more specific levels as well, such as per department or individual user. These aspects of the systems and methods can aid EMR and reporting tasks such as document summarization, similar document and finding key information to prepopulate a medical report.

FIG. 8B shows details for a language fine tuning module, according to an embodiment. FIG. 8B includes finetuning ecosystem 850, radiology reports 852 which includes radiology reports 812 from hospital-1 and other hospitals, pathology reports 854 which includes pathology reports 814 from hospital-1 and other hospitals, consult notes 856 which includes consult notes 816 from hospital-1 and other hospitals, language finetuning module 802, edit option finding module 804, clinical language models 806, and report/test type recommendations 860. FIG. 8B shows the value of distributed and/or crowdsourced information to help improve language models and edit recommendations. This allows clinicians, oncologists, nurses, assistance, physicians, and others in the medical system to all provide helpful feedback to the system for improving language models and edit recommendations in intelligent medical editors. Further, a model can be fine-tuned not just on the users of the system or requirements from the hospital or government, but also based on the patient. For example, fine tuning may be based on the timeline history of patients, type of disease, type of treatment, equipment use, medication dosages, and other factors along a patient journey. To effectively aid the aid the recommendations, multiple models could be aggregated, and recommendations could be based on the certain criterion/domain specific rules.

In FIG. 8B, finetuning ecosystem 850 exists to collect similar types of medical materials from different medical institutions, such as hospital-1, hospital-2, and hospital-3. Through this information, language model finetuning module can create models based on trends across geographies, medical systems, and different sizes of medical systems, all while developing a model specific to the type of medical materials involved. Thus, language models and edit recommendations can be specific and predictive to the report/test type, as shown in report/test type recommendations 860. For example, terminology in an oncology care pathway may be different from a cardiology care pathway.

FIG. 12 shows a clinical care guidelines deviation flow, according to an embodiment. FIG. 12 also relates to how clinical guidelines 326 may interact in the system, including with contradiction and incompleteness detection module 312. FIG. 12 includes clinical guidelines workflow 1200, clinical guidelines 1202, and deviation 1204. Clinical guidelines 1202 in this example are clinical assessment pretreatment evaluations which may suggest initial treatments. Certain tests may be run for a patient and based on the results of the test, certain treatments are recommended and/or required. If a user selects a treatment, medical examination, and/or medication that is non-standard in clinical guidelines 326 this may be flagged as a deviation 1204. The intelligent medical reporting tool 302 may display an alert of deviation and suggest alternative recommendations. One example of this with relation to FIG. 13 is contradiction notice 1322. A recommendation may be that ‘the guidelines have suggested this initial treatment’. Further, if different countries have different guidelines, the intelligent medical reporting tool can provide the correct guideline for this location/hospital, and also provide suggestions of how others may be doing in other hospitals/around the world.

Such a system can improve care and reduce multiple decisions that can lead to variations in medical care. Thus, such a system can save costs and provide better care to patients. Unwarranted deviations in care leads to suboptimal care, cost and affects quality of life. Clinical standardization remains on the top of the cost saving opportunities for CFOs while maintaining quality. Cancer guidelines and pathways guide physicians to render care that's clinically optimal and cost efficient through data and flowcharts. High complexity and multiple treatment possibilities of the cancer disease makes this very challenging to adhere to and leads to widespread deviations. An example for lung cancer guidelines shown in FIG. 12 illustrates the numerous decision points and data needed to traverse the guideline to the relevant point.

If a deviation from a guideline is approved by the user, the system can save the deviations for reporting and analysis. And then in the future, the system can check if a current deviation is consistent with deviations in the past for the same type of patient/situation, and provide a percentage confidence of previous deviations. The intelligent medical reporting tool 302 can also suggest what have the approved deviations in the past on a similar patient. Further, the system can provide statistics related to deviations, such as how many deviations are there for the clinician. And in some circumstances, if there is consensus across a medical field, the guidelines may be changed based on collective deviations that are good for healthcare.

FIG. 13 shows an example graphical user interface 1302 of implementation of the intelligent medical reporting tool 302 with suggested edits for a digital document according to an embodiment. In this example, the graphical user interface 1302 depicts a marked-up version of a medical report/record as marked-up by the intelligent medical reporting tool 302. The marked-up version includes event date recommendation data 1304, internal conflict recommendation data 1306, date recommendation data 1308, missing information recommendation data 1310, formatting recommendation data 1312, spelling recommendation data 1314, acronym expansion recommendation data 1316, image source recommendation data 1318, ontology code linking recommendation data 1320, contradiction notice data 1322, and diagnosis recommendation data 1324.

The user interface 1302 shows a medical report that already has some content. This content may have been typed in by a user, prepopulated and/or auto-generated by the intelligent medical reporting tool 302, or a combination of the two. The report has headings for different sections of the report that are shown in capital letters on the left. The intelligent medical reporting tool 302 reviews the material in the report and can provide alerts, notices, and recommendations to help improve the content of the report. These alerts, notices, and recommendations may be provided in different ways. FIG. 13 shows the alerts, notices, and recommendations as overlaid above the relevant section of the report. Alternatively, these may be little alert bubbles or icons that are clicked on for more information. Alternatively, these may be in a side bar or top bar of a user interface window. Alternatively, these may be through an interactive chatbot or conversation style interaction. These may be supported by generative artificial intelligence and large language models. There may be other ways of alerting and interacting such as vibration, touch, voice, and other audio inputs. The alerts, notices, and recommendations may also have a feedback button or icon. This allows a user to indicate errors, issues, suggestions, or positive feedback on the alert, notice, or recommendation. The intelligent medical reporting tool 302 can use this to improve its language models and edit recommendations.

The event date recommendation data 1304 is an example of adding additional information to the report that helps provide full detail. In the example, intelligent medical reporting tool 302 recommends adding “on Aug. 23^rd, 2022” to the report referring to the day the patient presented their medical complaints. This improves precision via event date recommendation 1304 using a patient's past data. This may be done through a patient profile as further discussed with reference to FIG. 19. The intelligent medical reporting tool 302 may detect a reference to a past event in a report based on a language model that looks at past tense words, tracks the patient journey, and/or certain fields in an EMR, as examples. If more information is available about the past event, the edit recommendation module 308 may make a recommendation like event date recommendation data 1304. More information about the past visit could be suggested, and both a short recommendation and long recommendation may be provided. And, based on the interactions of the users (which selections are made most often), the intelligent medical reporting tool 302 can prioritize the recommendations user's most prefer. Date recommendation data 1308 also suggests adding a date related to a recent medical test, in this case a mammogram.

The internal conflict recommendation data 1306 may use the rest of the report or other patient profile information to detect conflicts. In this example, the report in the fourth line states the “left breast.” And thus, in the third line where it says, “right breast”, the intelligent medical reporting tool 302 (for example through a contradiction and incompleteness detection module 312) can detect internal inconsistencies and ask a user for clarification. If a user clicks on the internal conflict recommendation data 1306 which is “left?” in this case, the intelligent medical reporting tool 302 can automatically change the word right into left. Another internal conflict example is the wrong date listed in a report, such as those related to event date recommendations.

The missing information recommendation data 1310 may provide recommendations for additions to medical materials. This would be helpful information that may be added to the report but currently does not exist. In the example of FIG. 13, a STAGE field of a medical report only says, “Stage 1.” Stage refers to the extent of cancer, such as how large the tumor is and if it has spread. The missing information recommendation data 1310 suggests it say TNM STAGE: T1, N0, and M0. The T refers to the size and extent of the main tumor. The main tumor is usually called the primary tumor. The N refers to the number of nearby lymph nodes that have cancer. The M refers to whether the cancer has metastasized. This means that the cancer has spread from the primary tumor to other parts of the body. Thus, with the addition of the missing information recommended, the report is more accurate, clear, and helpful to an oncologist reviewing the cancer situation.

The formatting recommendation data 1312 highlights ways in which formatting can be improved in the medical report. For example, certain report types are required or recommended to be in certain formats. In other examples the language model can detect that a report has a custom format and edit recommendation module can recommend edits related to maintaining the custom format. In the example of FIG. 13, all of the other report headings are in capital letters, and thus the medical editor 1300 recommends changing “Recommendations” to “RECOMMENDATIONS.” This improves readability by recommending formatting variations. Other types of formatting, spelling, and grammar edits may also be recommended using related modules. Certain types of spelling may be specific to medical ontologies and medical lexicography. Spelling recommendation data 1314 notes a misspelling of a specific drug (not in common dictionaries) and provides a corrected spelling recommendation. Having medical specific dictionaries, guidelines, and codes is useful. In addition, certain types of UB-04 and CPT codes could be in EMR records and the intelligent medical reporting tool 302 can detect inaccuracies, incorrectness, and misnaming issues. The Uniform Billing Form, known either as the UB-04 or CMS 1450, is a key player in the healthcare billing process. Further, if a report has a synoptic reporting standard, the system can identify parts of a report that are not following the synoptic reporting standard. Synoptic reports present information as discrete data elements and associated responses.

The acronym expansion recommendation data 1316 helps standard communication in medical reports. When a certain user uses a non-common acronym (or one that is better at complete words), an edit recommendation module can suggest that an acronym be fully stated. These may come from medical dictionaries or medical ontologies. This is helpful and more precise if a report or record will be viewed by patients or other non-medical experts who may not know all the acronyms that medical experts may use. In the example of FIG. 13, the report has the phrase “UOQ of the left breast.” Acronym expansion recommendation 1316 recommends changing “UOQ” to “upper outer quadrant.”

The image source recommendation data 1318 can provide additional source information about content. This may be the internet source of content with a hyperlink, it may be the source location in an imaging database such as a PACS (Picture Archiving and Communication System), or a source location in a patient profile or medical record. By adding a link or other source information, a person reading the report is better equipped to look into the details of the situation and make better decisions in a faster way. In the example of FIG. 13, the mammogram image of the left breast showed a mass in the upper outer quadrant. Image source recommendation 1318 recommends adding in the exact series and images related to the mammogram. Image source recommendation 1318 may also have a clickable link for a viewer to see the exact images. This improves completeness via image source recommendation using a patient's past radiology image data.

Ontology code linking recommendation data 1320 uses medical ontologies to improve precision in a medical report. In the example of FIG. 13, the intelligent medical reporting tool 302 recommends adding ICD codes 147.9 and 174.4 to the report. International Classification of Diseases (ICD) is a system used by physicians to classify and code all diagnoses, symptoms and procedures for claims processing. For example, ICD-9 code 174.9 for malignant neoplasm of breast (female) unspecified site is a medical classification as listed by WHO under the range. This is valuable to add to the report of FIG. 13 because it is in the specific section of the report of diagnosis. So having all the proper terminology and codes for the diagnosis is valuable.

The contradiction notice data 1322 identifies a contradiction of a recommended treatment compared to clinical guidelines (as discussed with respect to FIG. 12) and improves precision via finding contradictions with current medical knowledge using clinical guidelines. In the example of FIG. 13, the intelligent medical reporting tool 302 has detected that the treatment recommended contradicts with the NCCN (National Comprehensive Cancer Network) BINV-1 guidelines around breast cancer treatments and recommends adjustments to the recommended treatment plan. In some embodiments, if the user overrides this suggestion, the system may ask for more information as to the reasoning of the medical professional to document why they deviated from the clinical guidelines. Further, the diagnosis recommendation 1324 also recommends edits to the tests ordered for a patient, adding in a liver function panel (LFT) test based on clinical guidelines. This can save time by getting all the relevant medical testing done at the right times.

FIG. 16 shows another example user interface 1600 of the intelligent medical reporting tool 302 with suggested edits to a digital document, according to an embodiment. FIG. 16 shows medical record data 1602 that includes information about a patient. The medical record data can correspond to data that was entered by a user and/or auto-generated. In this example, the clinical information extraction module 304, for example, has determines that “lung” is a body part 104 and medicine name is a medicine that can be compared by the contradiction and incompleteness detection module 312 to verify it is a correct medicine name 1608. This is done through a medical spell correction popup recommendation and annotation popup, which is different than the above line recommendations shown in FIG. 13. Thus, FIG. 16 shows an example implementation of the intelligent medical reporting tool 302 that uses specific domain knowledge about the medical field and enables auto annotations and suggestions.

FIG. 18 shows an example process 1800 related to edit recommendations leveraging predictive and generative models, according to an embodiment. FIG. 18 shows components that can be included with the intelligent medical reporting tool 302 the edit recommendations module 308 in various embodiments. FIG. 18 includes edit recommendations 1802, current EMR report and details 1804, medical inputs 1806, finetuned module 1808, patient data 1810, prompt selection 1812, predictive models 1814, generative models 1816, grounding module 1820, and edit option finding module 1822. In accordance with process 1800, input into the intelligent medical reporting tool 302 and/or the edit recommendations module 308 includes medical inputs 1806 that provide a baseline for the grounding module, medical ontologies, medical dictionaries and other medical inputs. In the example of FIG. 18, medical input 1806 includes MeSH tree structure data for medical conditions. The Medical Subject Headings (MeSH) thesaurus is a controlled and hierarchically-organized vocabulary produced by the National Library of Medicine. It is used for indexing, cataloging, and searching of biomedical and health-related information. Input into the intelligent medical reporting tool 302 the edit recommendations module 308 also includes patent data from EMR reports and details 1804 as well as patient data 1810 including timeline metadata, such as from the internal hosted data within a hospital system.

In accordance with process 1800, prompt selection 1812 selects the most relevant information from the multiple options of data suitable for the current report. The finetuned module 1808 receives clinical language predictive models (performing tasks like classification and extraction) and fine tunes them based on feedback and usage information from within the system, and provides finetuned models to grounding module 1820. The finetuned module 1808 picks task specific models based on the current activity of the intelligent medical editor and its users. This may mean that the finetuned module 1808 picks models specific to the hospital, a disease, or a patient, in various examples. The fine-tuning process helps the model access external knowledge sources and improve its performance.

The grounding module 1820 takes in fine tuned models from the finetuned module 1808, selected information and data from prompt selection 1812, clinical language generative models 1816, metadata, ontologies, clinicians, and other system inputs. The grounding module 1820 determines factual information in the content it generates. This reduces hallucination and inaccuracy issues, and helps minimize randomness that may come from GAI usages. Clinical language generative models 1816 are generative artificial intelligence (GAI), generative language models (GLMs), foundational models, and large language models (LLMs) that perform tasks such as summary generation, text generation, conversational agents, among other tasks. Artificial intelligence (AI) and generative language models (GLMs) present significant opportunities for enhancing medical education, including the provision of realistic simulations, digital patients, personalized feedback, evaluation methods, and the elimination of language barriers. Clinical language generative models 1816 have been specifically trained in the clinical domain.

The edit option finding module 1822 receives fine tuned models and the relevant output from grounding module 1820 to predict options for edits to a medical record, and outputs edit recommendations. The edit recommendations 1802 can include providing recommendations, suggesting corrections, and providing alternate text in real time while a report is being types (as shown in FIG. 13 and other figures).

FIG. 19 shows a flow chart and block diagram related to patient profiling, according to an embodiment. Understanding information regarding a patient is important to the intelligent medical reporting tool 302. FIG. 19 includes patient profiling system 1900 which can correspond to a subsystem of system 300. The patient profiling system 1900 includes patient profiler 1902, similar patient finding 1904, intelligent medical reporting tool 302, future use evaluation 1908, finetuned model selection module 1910, patient data 1912, and past patient data 1914. The patient profiling system 1900 helps gather and understand information about patients.

The patient profiler 1902 builds a context aware patient profile based on patient data throughout the medical systems. This includes a patient EMR data and past TX (treatment/therapy) and DX (diagnosis) related to past patient data 1914. Patient data input into patient profiler also includes patient data 1912 from throughout the medical system including family history, past medical history, examination information, genomic information, drugs, social history, and physical activities. A patient profile can include information such as notable health changes that occur while a patient is at home, doctor or hospital visits, surgical procedures and any related complications, medications, immunizations, like the flu vaccine or tetanus shot, and other related health information. In addition, a user's preferences when working with the medical system technologies, such as a medical system app on their smartphones, an intelligent medical editor, or other computing devices they interact with. Further, patient data can include informal information that can be input via voice, video, personal wearable sensors, and other newer types of input (comments, perspectives, location data and spoken words—from smartphone, security cameras, and the like) as well as observations from friends and other third parties (such as did the patient take the medicine). Patient profiler 1902 is routinely updated to maintain context and history of the patient.

The intelligent medical reporting tool 302 can use patient profiles for edit recommendations, contradiction detection, and other aspects discussed throughout. For example, the intelligent medical reporting tool 302 may provide suggestions and edit recommendations to a user generating or editing a medical report/record using patient profile information as well as information related to similar patients (such as through similar patient finding 1904), similar reports, and similar diseases. Patient profiles and similar patient information can be absorbed (intake) into modules throughout the system as part of future use evaluation 1908, such as finetuned model selection module 1910.

FIG. 20A shows a flow chart for updating a medical report based on selected suggested edits, according to an embodiment. Medical process 2000A may include one or more of six steps that may be performed by the intelligent medical reporting tool 302 and/or system 300. In step 2002, the system (e.g., system 300) may identify a plurality of textual entries in a free text medical report. In step 2004, the system may generate suggestions with smart modules, such as detailed herein throughout and with examples at least related to FIG. 3. In step 2006, the system may provide suggested content related to the patient and medical report type. In step 2008, as a user is interacting with the report, the system may provide suggestions (e.g., edit recommendations) to the user related to the textual entries and other report content. Examples of this are shown in FIGS. 13-17 as examples in various embodiments. In step 2010, the system may provide warnings about probable errors (user or system generated errors) based on patient history, clinical care guidelines, medical ontologies, and other inputs and models. In step 2012, the system may update the medical report based on the chosen selections to the suggested content. Further, the system may update its learning modules (those that allow feedback input based on rules engines, artificial intelligence, generative artificial intelligence, machine learning, deep learning, and the like).

FIG. 20B shows a flow diagram of an example computer-implemented method 2000B for generating a medical report using an intelligent medical reporting tool, according to an embodiment. In accordance with method 2000B, at 2020, method 2000B comprises identifying, by a system comprising at least one processor (e.g., system 300 and other described herein), a textual entities in a medical report associated with a patient in association with receiving user input entering at least some of the textual entities into the report via a graphical user interface. At 2022, method 2000B comprises determining, by the system, edit recommendations related to one or more of the textual entities, where the edit recommendations are generated by an artificial intelligence model, and wherein the edit recommendation relate to possible errors or improvements based on information comprising medical history data of the patient, one or more clinical guidelines applicable to the report, and one or more medical ontologies. At 2024, method 2000B comprises providing, by the system, feedback information regarding the edit recommendations via the graphical user interface. At 2026, method 2000B comprises receiving, by the system, user feedback regarding selections related to the edit recommendations. At 2028, method 2000B comprises retraining the artificial intelligence model using the report and the user feedback as a training data, resulting in an updated version of the artificial intelligence model.

In this regard, the systems and methods disclosed herein allow for generating medical materials that are correct, timely, and adhere to applicable clinical guidelines and ontologies. The disclosed intelligent medical reporting tool 302 can leverage current government guidance and industry trends in improving healthcare. The intelligent medical reporting tool 302 can provide single solution to work with multi-modal time spaced data sources, use of patient histories and by-product modules providing processed data that can be leveraged for other downstream solutions, insights from clinician interactions to assist better with time and provide similar patient profiles for inpatients by using the vast patient histories. Overall patients benefit from previous cases and customized care, clinicians get better assistance and save more time to focus on the treatment and the hospitals benefit by getting one solution for EMR rather than using multiple tools for handling different data, reporting standards, etc. All this is done while maintaining the patient privacy.

The intelligent medical reporting tool 302 and associated systems disclosed herein help ensure the correctness, completeness and clarity of EMRs and other medical records, which improves clinical outcomes. Imprecise and incomplete EMRs adversely affect the speed and quality of all the subsequent clinical decisions such as diagnosis/treatment decisions, clinical trial selection, referencing guidelines and case studies. The intelligent medical reporting tool 302 helps by providing in-place AI assistance which provide editorial feedback and improvement suggestions to clinician while they are creating and updating an EMR.

The disclosed intelligent medical reporting tool 302 and associated systems ensure that the AI models are performing adequately to meet clinical standards is crucial before integrating them with a critical clinical system like EMR. High volume, variety and veracity of manual feedback is essential for validating and continuously improving the AI models. The two roadblocks which hinder such a collection of feedback are (1) the high time and cost of high-quality manual feedback, (2) the data privacy requirements which result in administrative and data anonymization overheads. The intelligent medical reporting tool 302 helps by (1) creating a low-intrusion way for clinicians to provide in-place feedback to AI as well as by (2) creating a federated method to improve the AI models which eliminate the need to share patient data outside of the organization/legal boundaries, thereby enabling wide crowdsourcing of training/validation data for AI models.

In addition, the intelligent medical reporting tool 302 and associated systems help users by pre-annotating a report accurately before clinician opens it and/or during report generating and presents the annotation data in a readily comprehensible and easily modifiable view to the clinician. Both the annotations (including prepopulated information) and the clinician modified/created annotations help structurally enrich the report with codified concepts. This has following clinical benefits, bring much needed standardization to clinical report. It can be difficult for other users to quickly read other doctor's reports due to them not following synoptic reporting standards required for oncology. Real-time autocorrections and suggestions coming out of NLP subsystem helps in this. The intelligent medical reporting tool 302 is particularly useful as it auto-fills or auto-generates content based on concepts related to medical standards as mentioned throughout (including FHIR, UB-04 and the like). If the codified concepts can be parsed directly from report, with integration with EMR systems, systems and methods herein can autofill such data, which is a much needed feature both for productivity of doctors as well as enormous benefit to downstream applications.

The systems and processes described below can be embodied within hardware, such as a single integrated circuit (IC) chip, multiple ICs, an application specific integrated circuit (ASIC), or the like. Further, the order in which some or all of the process blocks appear in each process should not be deemed limiting. Rather, it should be understood that some of the process blocks can be executed in a variety of orders, not all of which may be explicitly illustrated in this disclosure.

The illustrated aspects of the disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in both local and remote memory storage devices.

Moreover, it is to be appreciated that various components described in this description can include electrical circuit(s) that can include components and circuitry elements of suitable value in order to implement the embodiments of the subject innovation(s). Furthermore, it can be appreciated that many of the various components can be implemented on one or more integrated circuit (IC) chips. For example, in one embodiment, a set of components can be implemented in a single IC chip. In other embodiments, one or more of respective components are fabricated or implemented on separate IC chips.

Referring to FIG. 21, there is illustrated a schematic block diagram of a computing environment 2100 in accordance with this disclosure in which the subject systems, methods and computer readable media can be deployed. The computing environment 2100 includes one or more client(s) 2102 (e.g., laptops, smart phones, PDAs, media players, computers, portable electronic devices, tablets, and the like). The client(s) 2102 can be hardware and/or software (e.g., threads, processes, computing devices). The computing environment 2100 also includes one or more server(s) 2104. The server(s) 2104 can also be hardware or hardware in combination with software (e.g., threads, processes, computing devices). The servers 2104 can house threads to perform transformations by employing aspects of this disclosure, for example. In various embodiments, one or more of the subject front end-components can be deployed as hardware and/or software at a client 2102 and one or more of the subject back-end components can be deployed as hardware and/or software at server 2104. One possible communication between a client 2102 and a server 2104 can be in the form of a data packet transmitted between two or more computer processes wherein the data packet may include video data. The data packet can include a metadata, e.g., associated contextual information, for example. The computing environment 2100 includes a communication framework 2106 (e.g., a global communication network such as the Internet, or mobile network(s)) that can be employed to facilitate communications between the client(s) 2102 and the server(s) 2104.

Communications can be facilitated via a wired (including optical fiber) and/or wireless technology. The client(s) 2102 include or are operatively connected to one or more client data store(s) 2108 that can be employed to store information local to the client(s) 2102 (e.g., associated contextual information). Similarly, the server(s) 2104 are operatively include or are operatively connected to one or more server data store(s) 2110 that can be employed to store information local to the servers 2104.

In one embodiment, a client 2102 can transfer an encoded file, in accordance with the disclosed subject matter, to server 2104. Server 2104 can store the file, decode the file, or transmit the file to another client 2102. It is to be appreciated, that a client 2102 can also transfer uncompressed file to a server 2104 and server 2104 can compress the file in accordance with the disclosed subject matter. Likewise, server 2104 can encode video information and transmit the information via communication framework 2106 to one or more clients 2102.

FIG. 22 illustrates a schematic block diagram of another example computing environment 2200 in accordance with this disclosure in which the subject systems, methods and computer readable media can be deployed. The computing environment 2200 includes a cloud deployment architecture consisting of one or more clients 2202 that can be communicatively coupled to a system cloud 2204 via a network (e.g., the Internet). The system cloud 2204 can include a cloud load balances, one or more application container, one or more cloud service containers, a cloud data store, and a cloud network that communicatively couples the one or more cloud components to the cloud data store. In accordance with the cloud deployment architecture, the clients 2202 can include one or more clients devices (e.g., a mobile device, a laptop computer, a desktop computer, etc.) which can include or employ a suitable application (e.g., a native mobile application, a web-based application, a thin/thick client application, etc.) to access and employ one or more features and functionalities of the subject native/reconstructed medical imaging systems deployed in the system cloud 2204. In various implementations, the one or more components of system 100 can be distributed between the clients 2202 and the system cloud 2204.

FIG. 23 illustrates a schematic block diagram of another example computing environment 2300 in accordance with this disclosure in which the subject systems, methods and computer readable media can be deployed. The computing environment 2300 includes a virtualized enterprise deployment consisting of one or more clients 2202 that can be communicatively coupled to a remote data center 2302 via a network (e.g., the Internet). The remote data center 2302 can include an application servers subnet 2304 which can provide a load balancer, one or more application containers, one or more virtualized servers and one or more rack servers. The data center 2302 can also include one or more data stores that can be communicatively coupled to the application servers subnet 2304 via a data center network. In accordance with the virtualized enterprise deployment, the clients 2202 can include one or more clients devices (e.g., a mobile device, a laptop computer, a desktop computer, etc.) which can include or employ a suitable application (e.g., a native mobile application, a web-based application, a thin/thick client application, etc.) to access and employ one or more features and functionalities of the subject native/reconstructed medical imaging systems deployed in the data center 2302 and application servers subnet 2304. In various implementations, the one or more components of systems 100 can be distributed between the clients 2202 and the application servers subnet 2304 and the one or more data stores can be provided remotely at the data center 2302.

FIG. 24 illustrates a schematic block diagram of another example computing environment 2400 in accordance with this disclosure in which the subject systems, methods and computer readable media can be deployed. The computing environment 2400 includes a local enterprise deployment consisting of one or more clients 2202 that can be communicatively coupled to an application servers subnet 2404 via a network (e.g., the Internet). In accordance with this embodiment, the application servers subnet 2404 can be provided at the enterprise premises 2402 (e.g., as opposed to a remote data center 2302). The application servers subnet 2404 can include a load balancer, one or more application containers and one or more servers. The application servers subnet 2404 can be communicatively coupled to one or more data stores provided at the enterprise premises 2402 via an enterprise network. Similar to the cloud and virtualized enterprise deployments, the clients 2202 can include one or more clients devices (e.g., a mobile device, a laptop computer, a desktop computer, etc.) which can include or employ a suitable application (e.g., a native mobile application, a web-based application, a thin/thick client application, etc.) to access and employ one or more features and functionalities of the subject native/reconstructed medical imaging systems (e.g., system 100 and the like) deployed at the enterprise premises 2402 and the application servers subnet 2404. In various implementations, the one or more components of systems herein can be distributed between the clients 2202 and the application servers subnet 2404 and the one or more data stores can be provided at the enterprise premises 2402.

FIG. 25 illustrates a schematic block diagram of another example computing environment in accordance with this disclosure in which the subject systems, methods and computer readable media can be deployed. The computing environment includes a local device deployment in which all of the components of systems herein are provided at a single client device 2502. With this implementation, the client device 2502 can include a web-based application which can be communicatively coupled via a loopback to one or more application containers. The one or more application containers can be communicatively coupled via a loopback to one or more databases and/or one or more local file systems.

With reference to FIG. 26, a suitable environment 2600 for implementing various aspects of the claimed subject matter includes a computer 2602. The computer 2602 includes a processing unit 2604, a system memory 2606, a codec 2605, and a system bus 2608. The system bus 2608 couples system components including, but not limited to, the system memory 2606 to the processing unit 2604. The processing unit 2604 can be any of various available processors. Dual microprocessors and other multiprocessor architectures also can be employed as the processing unit 2604.

The system bus 2608 can be any of several types of bus structure(s) including the memory bus or memory controller, a peripheral bus or external bus, and/or a local bus using any variety of available bus architectures including, but not limited to, Industrial Standard Architecture (ISA), Micro-Channel Architecture (MSA), Extended ISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB), Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus (USB), Advanced Graphics Port (AGP), Personal Computer Memory Card International Association bus (PCMCIA), Firewire (IEEE 22104), and Small Computer Systems Interface (SCSI).

The system memory 2606 includes volatile memory 2610 and non-volatile memory 2612. The basic input/output system (BIOS), containing the basic routines to transfer information between elements within the computer 2602, such as during start-up, is stored in non-volatile memory 2612. In addition, according to present innovations, codec 2605 may include at least one of an encoder or decoder, wherein the at least one of an encoder or decoder may consist of hardware, a combination of hardware and software, or software. Although, codec 2605 is depicted as a separate component, codec 2605 may be contained within non-volatile memory 2612. By way of illustration, and not limitation, non-volatile memory 2612 can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory 2210 includes random access memory (RAM), which acts as external cache memory. According to present aspects, the volatile memory may store the write operation retry logic and the like. By way of illustration and not limitation, RAM is available in many forms such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), and enhanced SDRAM (ESDRAM).

Computer 2602 may also include removable/non-removable, volatile/non-volatile computer storage medium. FIG. 19 illustrates, for example, disk storage 2614. Disk storage 2614 includes, but is not limited to, devices like a magnetic disk drive, solid state disk (SSD) floppy disk drive, tape drive, Zip drive, flash memory card, or memory stick. In addition, disk storage 2614 can include storage medium separately or in combination with other storage medium including, but not limited to, an optical disk drive such as a compact disk ROM device (CD-ROM), CD recordable drive (CD-R Drive), CD rewritable drive (CD-RW Drive) or a digital versatile disk ROM drive (DVD-ROM). To facilitate connection of the disk storage devices 2614 to the system bus 2608, a removable or non-removable interface is typically used, such as interface 2616.

It is to be appreciated that FIG. 26 describes software that acts as an intermediary between users and the basic computer resources described in the suitable operating environment 2600. Such software includes an operating system 2618. Operating system 2618, which can be stored on disk storage 2614, acts to control and allocate resources of the computer system 2602. Applications 2620 take advantage of the management of resources by operating system 2618 through program modules 2624, and program data 2626, such as the boot/shutdown transaction table and the like, stored either in system memory 2606 or on disk storage 2614. It is to be appreciated that the claimed subject matter can be implemented with various operating systems or combinations of operating systems.

A user enters commands or information into the computer 2602 through input device(s) 2628. Input devices 2628 include, but are not limited to, a pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, joystick, game pad, satellite dish, scanner, TV tuner card, digital camera, digital video camera, web camera, microphone, and the like. These and other input devices connect to the processing unit 2604 through the system bus 2608 via interface port(s) 2630. Interface port(s) 2630 include, for example, a serial port, a parallel port, a game port, and a universal serial bus (USB). Output device(s) 2636 use some of the same type of ports as input device(s). Thus, for example, a USB port may be used to provide input to computer 2602, and to output information from computer 2602 to an output device 2636. Output adapter 2634 is provided to illustrate that there are some output devices 2636 like monitors, speakers, and printers, among other output devices 2636, which require special adapters. The output adapters 2634 include, by way of illustration and not limitation, video and sound cards that provide a means of connection between the output device 2636 and the system bus 2608. It should be noted that other devices and/or systems of devices provide both input and output capabilities such as remote computer(s) 2638.

Computer 2602 can operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 2638. The remote computer(s) 2638 can be a personal computer, a server, a router, a network PC, a workstation, a microprocessor based appliance, a peer device, a smart phone, a tablet, or other network node, and typically includes many of the elements described relative to computer 2602. For purposes of brevity, only a memory storage device 2640 is illustrated with remote computer(s) 2638. Remote computer(s) 2638 is logically connected to computer 2602 through a network interface 2642 and then connected via communication connection(s) 2644. Network interface 2642 encompasses wire and/or wireless communication networks such as local-area networks (LAN) and wide-area networks (WAN) and cellular networks. LAN technologies include Fiber Distributed Data Interface (FDDI), Copper Distributed Data Interface (CDDI), Ethernet, Token Ring and the like. WAN technologies include, but are not limited to, point-to-point links, circuit switching networks like Integrated Services Digital Networks (ISDN) and variations thereon, packet switching networks, and Digital Subscriber Lines (DSL).

Communication connection(s) 2644 refers to the hardware/software employed to connect the network interface 2642 to the bus 2608. While communication connection 2644 is shown for illustrative clarity inside computer 2602, it can also be external to computer 2602. The hardware/software necessary for connection to the network interface 2642 includes, for exemplary purposes only, internal and external technologies such as, modems including regular telephone grade modems, cable modems and DSL modems, ISDN adapters, and wired and wireless Ethernet cards, hubs, and routers.

What has been described above includes examples of the embodiments of the present invention. It is, of course, not possible to describe every conceivable combination of components or methodologies for purposes of describing the claimed subject matter, but it is to be appreciated that many further combinations and permutations of the subject innovation are possible. Accordingly, the claimed subject matter is intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims. Moreover, the above description of illustrated embodiments of the subject disclosure, including what is described in the Abstract, is not intended to be exhaustive or to limit the disclosed embodiments to the precise forms disclosed. While specific embodiments and examples are described in this disclosure for illustrative purposes, various modifications are possible that are considered within the scope of such embodiments and examples, as those skilled in the relevant art can recognize.

In particular and in regard to the various functions performed by the above described components, devices, circuits, systems and the like, the terms used to describe such components are intended to correspond, unless otherwise indicated, to any component which performs the specified function of the described component (e.g., a functional equivalent), even though not structurally equivalent to the disclosed structure, which performs the function in the disclosure illustrated exemplary aspects of the claimed subject matter. In this regard, it will also be recognized that the innovation includes a system as well as a computer-readable storage medium having computer-executable instructions for performing the acts and/or events of the various methods of the claimed subject matter.

The aforementioned systems/circuits/modules have been described with respect to interaction between several components/blocks. It can be appreciated that such systems/circuits and components/blocks can include those components or specified sub-components, some of the specified components or sub-components, and/or additional components, and according to various permutations and combinations of the foregoing. Sub-components can also be implemented as components communicatively coupled to other components rather than included within parent components (hierarchical). Additionally, it should be noted that one or more components may be combined into a single component providing aggregate functionality or divided into several separate sub-components, and any one or more middle layers, such as a management layer, may be provided to communicatively couple to such sub-components in order to provide integrated functionality. Any components described in this disclosure may also interact with one or more other components not specifically described in this disclosure but known by those of skill in the art.

In addition, while a particular feature of the subject innovation may have been disclosed with respect to only one of several implementations, such feature may be combined with one or more other features of the other implementations as may be desired and advantageous for any given or particular application. Furthermore, to the extent that the terms “includes,” “including,” “has,” “contains,” variants thereof, and other similar words are used in either the detailed description or the claims, these terms are intended to be inclusive in a manner similar to the term “comprising” as an open transition word without precluding any additional or other elements.

As used in this application, the terms “component,” “system,” or the like are generally intended to refer to a computer-related entity, either hardware (e.g., a circuit), a combination of hardware and software, software, or an entity related to an operational machine with one or more specific functionalities. For example, a component may be, but is not limited to being, a process running on a processor (e.g., digital signal processor), a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a controller and the controller can be a component. One or more components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers. Further, a “device” can come in the form of specially designed hardware; generalized hardware made specialized by the execution of software thereon that enables the hardware to perform specific function; software stored on a computer readable storage medium; software transmitted on a computer readable transmission medium; or a combination thereof.

Moreover, the words “example” or “exemplary” are used in this disclosure to mean serving as an example, instance, or illustration. Any aspect or design described in this disclosure as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the words “example” or “exemplary” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. That is, unless specified otherwise, or clear from context, “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then “X employs A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from context to be directed to a singular form.

Computing devices typically include a variety of media, which can include computer-readable storage media and/or communications media, in which these two terms are used in this description differently from one another as follows. Computer-readable storage media can be any available storage media that can be accessed by the computer, is typically of a non-transitory nature, and can include both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer-readable storage media can be implemented in connection with any method or technology for storage of information such as computer-readable instructions, program modules, structured data, or unstructured data. Computer-readable storage media can include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other tangible and/or non-transitory media which can be used to store desired information. Computer-readable storage media can be accessed by one or more local or remote computing devices, e.g., via access requests, queries or other data retrieval protocols, for a variety of operations with respect to the information stored by the medium.

In view of the exemplary systems described above, methodologies that may be implemented in accordance with the described subject matter will be better appreciated with reference to the flowcharts of the various figures. For simplicity of explanation, the methodologies are depicted and described as a series of acts. However, acts in accordance with this disclosure can occur in various orders and/or concurrently, and with other acts not presented and described in this disclosure. Furthermore, not all illustrated acts may be required to implement the methodologies in accordance with certain aspects of this disclosure. In addition, those skilled in the art will understand and appreciate that the methodologies could alternatively be represented as a series of interrelated states via a state diagram or events. Additionally, it should be appreciated that the methodologies disclosed in this disclosure are capable of being stored on an article of manufacture to facilitate transporting and transferring such methodologies to computing devices. The term article of manufacture, as used in this disclosure, is intended to encompass a computer program accessible from any computer-readable device or storage media

It is to be understood that the above description is intended to be illustrative, and not restrictive. For example, the above-described embodiments (and/or aspects thereof) may be used in combination with each other. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the various embodiments of the invention without departing from their scope. While the dimensions and types of materials described herein are intended to define the parameters of the various embodiments of the invention, the embodiments are by no means limiting and are exemplary embodiments. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. The scope of the various embodiments of the invention should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

In the appended claims, the terms “including” and “in which” are used as the plain-English equivalents of the respective terms “comprising” and “wherein.” Moreover, in the following claims, the terms “first,” “second,” and “third,” etc. are used merely as labels, and are not intended to impose numerical requirements on their objects. Further, the limitations of the following claims are not written in means-plus-function format and are not intended to be interpreted based on 35 U.S.C. § 112, sixth paragraph, unless and until such claim limitations expressly use the phrase “means for” followed by a statement of function void of further structure.

This written description uses examples to disclose the various embodiments of the invention, including the best mode, and also to enable any person skilled in the art to practice the various embodiments of the invention, including making and using any devices or systems and performing any incorporated methods. The patentable scope of the various embodiments of the invention is defined by the claims, and may include other examples that occur to those skilled in the art. Such other examples are intended to be within the scope of the claims if the examples have structural elements that do not differ from the literal language of the claims, or if the examples include equivalent structural elements with insubstantial differences from the literal languages of the claims. Embodiments of the present disclosure shown in the drawings and described above are example embodiments only and are not intended to limit the scope of the appended claims, including any equivalents as included within the scope of the claims. Various modifications are possible and will be readily apparent to the skilled person in the art. It is intended that any combination of non-mutually exclusive features described herein are within the scope of the present invention. That is, features of the described embodiments can be combined with any appropriate aspect described above and optional features of any one aspect can be combined with any other appropriate aspect. Similarly, features set forth in dependent claims can be combined with non-mutually exclusive features of other dependent claims, particularly where the dependent claims depend on the same independent claim. Single claim dependencies may have been used as practice in some jurisdictions require them, but this should not be taken to mean that the features in the dependent claims are mutually exclusive.