Method for Automatically Converting a Text string to an Interactive Video Experience

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No. 63/414,976, filed Oct. 11, 2022, the contents of which are incorporated by reference herein.

BACKGROUND

This invention relates to computer implemented training methodologies.

Geographically-distributed employees that have responsibilities that include face-to-face or live telephone-based customer interactions often may be challenged in consistently delivering approved corporate messaging with polished delivery that is appropriate and compelling for their customers. Many corporate training programs involve intensive “boot camp” type of engagements, i.e., typical practices such as geographically-distributed employees, e.g., a sales force travel to a common geographical location that is out of the sales field and where these employees are isolated for intensive training. Typically, these boot camp type training programs run for a finite time, conclude, and are often not repeated at least for the same topic under the assumption which may not be fully verified that the person has absorbed the information.

One use case involves interactive role play, where a computer simulates a virtual actor and a user carries on a conversation with the virtual actor. Prior techniques are resource intensive, e.g., computationally resource intensive, especially when there may be branching moments in the conversation.

SUMMARY

According to an aspect of the invention, a computer-implemented method includes receiving, by a computer, a text string having one or more branching moments, with the computer including a processor, memory, a non-transitory computer storage, and input/output devices, processing by the computer the text string to recognize indications of the one or more branching moments given the text string, converting, by the computer, the processed text string and the indications of the one or more branching moments into executable computer code, receiving, by the computer, a response to a given one of the converted one or more branching moments from a predetermined set of responses, generating, by the computer from the executable computer code and media elements for the response, a virtual respondent, and storing, by the computer, the executable computer code and the media elements as a file in the computer storage that represents the virtual respondent.

Other aspects include a data processing system and a computer program product tangibly storing a computer program on a non-transitory computer readable medium.

The following are some of the embodiments, amongst others disclosed herein, within the scope of one or more of the above aspects.

Execute the executable computer code to render the response to the text string at the one or more branching moments into computer generated audio, and sending the computer generated audio to a client device to cause the client device to present the computer generated audio at the one or more branching moments. Generate from the executable computer code and the media elements, a virtual actor and cause the virtual actor to render a selected response. Pause video of the virtual respondent, cause choice buttons, for the given one of the converted one or more branching moments, to be rendered in juxtaposition to the paused video of the virtual respondent, and receiving input indicating selection of one of the choice buttons which selection indicates the response. Generate a series of text only written responses and cause the series of text only written responses to be rendered for a selected response. Receive an audio signal encoding speech from a participant operating a client device and convert the received audio signal into the text string. The client device can be a separate device from the computer. Processing the text string to recognize indications of the one or more branching moments given the text string can include detecting, in the text string, the one or more branching moments. Converting the processed text string can include converting the processed text string and the indications of the one or more branching moments into JavaScript Object Notation (JSON). Generating, using the JSON and the media elements, the virtual respondent can include generating one or more files that reference the JSON and the media elements.

One or more of the following advantages may be provided by one or more of the above aspects.

One or more of the above aspects provides a user with a self-coaching training experience that can be checked by a user's manager, etc. By depicting a virtual respondent in a video with the human user in juxtaposition the user can be practicing his presentation. The virtual respondent is a computer generated video of a virtual actor that provides responses to the user's narrative in the form of a computer generated narrative of the virtual actor. Alternatively, the virtual respondent can be depicted only as text-based speech bubbles that show responses to the user's narrative, but without the computer generated actor video or audio. The computer generated narrative has branching moments that allow the conversation to branch in different directions depending on the selections made by the user. Other variations are possible.

In some implementations, the systems and methods described in this specification can improve over other systems, e.g., existing computer implemented training systems. For instance, the systems and methods described in this specification can reduce computational resource usage, e.g., by converting the processed text string and the indications of the one or more branching moments into executable computer code, generating or storing data for the virtual respondent, or a combination of both. In some implementations, the systems and methods described in this specification can enable automation for training that was not previously available, e.g., using executable code for branching moments. For instance, processing a text string to recognize indications of the one or more branching moments given the text string, converting a text string and indications of the one or more branching moments into executable computer code, or both, can enable automation that was previously unavailable.

The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a diagrammatic view of a networked system executing software for recognizing and converting branching moments in a text string into interactive videos.

FIG. 2 is a flow chart of processing for recognizing branching moments.

FIGS. 3-5 are flow charts of processes for generating the interactive videos.

FIGS. 6-9 are diagrams of screenshots of graphical user interfaces representing segments of an interactive video.

FIG. 10 is a block diagram of a data processing system.

DETAILED DESCRIPTION

Described below is an integrated information and communication platform that enables devices to produce video in part from parsing a text string that is uploaded to a server/database. In the platform 10, a text string includes text messages, etc.

Referring now to FIG. 1, an exemplary networked computer platform 10 “platform 10” that includes functionality for executing training software for converting a text string into an interactive video is shown. The platform 10 includes a computer server system (server) 12 for processing of one or more text strings sent from users 14 by client devices 14a. The text strings are processed and stored in a database, via server 12 that can be accessed by individuals, such a managers 16, via client devices 16a. The server 12 includes an application or a web browser acting as a client utilizing an instance of a hosted application on a web server.

Client devices 14a and 16a can be any combination of, e.g., personal digital assistants, cell phones, computer systems, media-player-type devices, tablet computers and so forth. The client devices 14a enable the users 14 to input and receive information as well as to upload video and audio and/or text to the server 12 for use by the managers 16 (and/or other users). The platform 10 also includes a database 27 containing configuration settings, information, and media, such as the text string.

In some embodiments, the platform 10 is implemented in a cloud-based environment for long-term storage and management of captured media and servers in the cloud have instances of the management software 30 execute on those servers 12 to analyze the captured media to generate useful metadata and previews to allow users to find specific media, and distinguish specific media from other similar media easily and reliably.

A network-capable portable computer system (such as a tablet device) includes an application for executing process 40 for employee practice and performance improvement. Alternatively, computer systems may utilize web browsing software to act as a client that utilizes an instance of a hosted version of the same/similar application functionality. Many such instances of these applications are used to interface with networked databases 27 that store information and media for the applications.

Referring now the FIG. 2, the server 12 executes a process 40 that receives 42 a text string and processes 44 the received text string. The received text string has one or more branching moments to recognize indications of the one or more branching moments. Branching moments are defined as pauses in a conversation where one party should or can react or respond to something that was said by the other participant(s) and the choice of the response could take the conversation in different directions. For instance, the received text string can represent a statement made by a participant operating a client device, e.g., a phrase spoken by the participant.

The process 40 converts 46 the processed text string and the indications of the one or more branching moments into executable computer code (executable computer instructions). The server 12 receives, e.g., selects, 48 a response to the given one of the converted one or more branching moments from a predetermined set of responses. The server 12 generates 50 from the executable computer code and media elements a convincing, virtual respondent “virtual respondent,” and stores 52 the executable computer code and media elements as a file in the computer storage.

Referring now to FIG. 3, process 60 is shown. Process 60 produces 62 low-level components, e.g., JSON (JavaScript Object Notation), media elements, or both. A template links 64 at least some of logic, e.g., on tab 2 of a spreadsheet. The logic can be a script in the spreadsheet. The template converts the logic to the JSON (JavaScript Object Notation) language so that it will be machine-readable by JavaScript code. Separately, the process captures 66 the text, e.g., from all cells from tab 1, column B, “Virtual Actor Statement” in the spreadsheet, and processes it with a video generation service, e.g., to generate one or more media elements. Either a human intervention or an automated process can be used to capture and process the text Virtual Actor Statements to generate the media elements such as videos. The result is a series of videos, each named for the “Step Number” in tab 1, column A, e.g., in the spreadsheet. Either a human intervention or an automated process can be used to name the videos.

Process 60 next converts 70 the low-level components into a SCORM (Sharable Content Object Reference Model) e-learning module. SCORM is a collection of standards and specifications for web-based electronic educational technology (also called e-learning). SCORM defines communications between client-side content and a host system (called “the run-time environment”), which is commonly supported by a learning management system. SCORM also defines how content may be packaged into a transferable ZIP file called “Package Interchange Format.” (See en.wikipedia.org/wiki/Sharable_Content_Object_Reference_Modelen.wikipedia.org/wiki/Sharable_Content_Object_Reference_Model for more information.)

Process 60 has a pre-produced folder structure, produced by copying a standard set of folders and files. Process 60 integrates 72 the JSON and video files, in order to make a valid SCORM file. Process 60 places 74 the videos in a subfolder within that structure, and adds the file names of the videos to the “manifest” file, in order to make it SCORM compliant, and pastes 76 the JSON from step 1 into a pre-existing “index.html” file that contains the logic to interpret JSON and present it as the interactive adaptive conversational experience for the “virtual respondent,” as a computer generated video of a virtual actor. (Code snippets appear below.) Process 60 zips 78 the entire folder structure into a single ZIP file that is now a valid SCORM e-learning module.

Referring now to FIG. 4, process 80 is shown. Process 80 incorporates 82 the SCORM e-learning module into a split-pane video practice exercise, native to the platform 10. Inside the platform 10, process 80 imports 84 the SCORM file, which provides the imported SCORM file as a platform content item and generates a unique Content ID etc. In the platform 10, process 80 is responsive 86 to point-and-click user type interfaces to produce a new video practice exercise. Process 80 specifies 88 a “Dialog Simulator” type of exercise that uses a split-screen recorder. Process 80 prompts 90 the trainer to specify which content item to use for the left side, for which the user selects 92 the SCORM e-learning module that was imported. The video practice exercise can then be assigned to specific learners or incorporated into courses and curriculums for assigning to waves of learners.

The code samples below are relevant to the SCORM e-learning module, i.e., the process that was described in the prior paragraph.

Referring now to FIG. 5, a process 100 that uses the JavaScript, CSS, and HTML code within index.html to generate 102 a set of HTML <video> elements that instruct a web browser to display each of the virtual actor statements in the right sequence in reaction to choices made by the human participant, e.g., which choices can represent selection of a branching moment from multiple branching moments. All video elements are made invisible by default, and paused at their beginnings. The JavaScript and CSS (Cascading Style Sheets) code makes the very first video, for step 1 in the template, visible, but not yet playing, and displays a “Begin” button atop of the first video. (CSS is a style sheet language used for describing a presentation of a document written in a markup language such as HTML or XML (including XML dialects such as SVG, MathML or XHTML). (See en.wikipedia.org/wiki/CSS.)

Once the user clicks “Begin”, the process 100 instructs 104 the webpage to display “choice buttons” (indications of branching moments) for the first step in the conversation at the end of playback, hide the “begin button” and start playing the first video. When video playback ends, the choice buttons (indicated branching moments) are shown to the user. The user can only click one of the choices at any given step in the conversation. That choice determines which video the platform 10 will play next. The process 100 repeats 106 those steps based on the user's choice. The process instructs 108 the webpage to display the choice buttons (in randomized order) for the selected step in the conversation at the end of playback, hide 110 the previous choice buttons, and start playing 112 the chosen video. The process 100 can repeat the instruct webpage step 104 until the user reaches a conversation step that offers only one choice, “End.” At that point, the process 100 displays a message indicating the end of the conversation.

All of that is achieved with the following pseudo code inside the index.html file.

Referring now to FIGS. 6-8, depictions of a virtual respondent, as a video and a human that are in juxtaposition and that are practicing a presentation are shown. In this implementation, the virtual respondent is a computer generated video of a virtual actor that provides responses to the user's narrative in the form of computer generated narrative of the virtual actor. In other implementations, the virtual respondent is depicted only as text-based speech bubbles that show responses to the user's narrative, but without the computer generated actor video or audio or both. Other variations are possible.

FIG. 6 shows an initial screen shot with a “begin” button placed over a portion of the virtual actor. The human uses his mouse to start the presentation by selecting the “begin” button.

FIG. 7 shows a subsequent screen shot later in the presentation, with a “May there be another cause?,” “How can I help?” and “Can I have your full name?” statements that present choices to the human participant in the form of buttons generated over a portion of the virtual actor.

FIG. 8 shows a subsequent screen shot still later in the presentation, with a “Do you have your order number?” and “What went wrong?” statements that present choices to the human participant in the form of buttons generated over a portion of the virtual actor.

FIG. 9 shows a subsequent screen shot still later in the presentation, with a “I'm sorry. Your business is important to us, so I'll see if we can find a resolution” statement that presents a choice to the human participant in the form of a button generated over a portion of the virtual actor.

An example conversation is shown below in Table 1 through Table 4, which are partitions taken from a master table having the following columns.

• • Step Number;
  • VAS;
  • Choice 1;
  • Go to 1;
  • Choice 2;
  • Go to 2;
  • Choice 3;
  • Go to 3;
  • Choice 4;
  • Go to 4; and
  • Notes;
    where “VAS” corresponds to a “Virtual Actor Statement,” there being one video for each virtual actor statement.

Table 1 below shows a step number and a virtual actor statement for a sample conversation simulation script for “Order Not Delivered.” Table 1 also shows the step number and a choice 1 with a go to for choice 1 for the respective virtual actor statements. In Table 1 through Table 4, the choice numbers correspond to different branches for a branching moment from one or more branching moments, e.g., a particular selected branch.

Table 2 below shows the step number and a choice 2 with a go to for choice 2 for the respective virtual actor statements from Table 1, above, e.g., to the extent that the branching moments include a second branch. Table 2 also shows the step number and a choice 3 with a go to for choice 3 for the respective virtual actor statements from Table 1, above, e.g., to the extent that the branching moments include a third branch.

Table 3 below shows the step number and a choice 4 with a go to for choice 4 for the respective virtual actor statements from Table 1, above, e.g., to the extent that the branching moments include a fourth branch. Table 3 also shows the step number and Notes, if any, indicating a disposition (steps 6.1 through 6.5 of the conversation for the respective virtual actor statements from Table 1, above.)

Tables 1-3 can be configured to render conversations from very positive to very negative, by selecting different initial starting statements. The script author will have some conversational paths that end well, with a happy customer, and other conversational paths that could end badly, with an irate customer escalating to a manager or hanging up on the rep. The choices that the rep makes along the way determine which path the conversation takes and the resulting outcome at the end.

Table 4 below shows the statements (col. 1), the statements ready to be copied/pasted as JavaScript and the statements ready to be copied/pasted as XML for a Manifest file (only for video).

As shown in FIG. 10, the essential elements of a computer are one or more programmable processors for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled, (via bus, fabric, network, etc.) to I/O interfaces, network/communication subsystems, and one or more mass storage devices for storing data (e.g., magnetic, magneto optical disks, or optical disks).

Embodiments can be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations thereof. Embodiments can be implemented in a computer program product tangibly stored in a machine-readable (e.g., computer readable) hardware storage device for execution by a programmable processor; and method actions can be performed by a programmable processor executing a program of executable computer code (executable computer instructions) to perform functions of the invention by operating on input data and generating output. Embodiments can be implemented advantageously in one or more computer programs executable on a programmable system, such as a data processing system that includes at least one programmable processor coupled to receive data and executable computer code from, and to transmit data and executable computer code to, memory, and a data storage system, at least one input device, and at least one output device. Each computer program can be implemented in a high-level procedural or object oriented programming language, or in assembly or machine language if desired; and in any case, the language can be a compiled or interpreted language.

Suitable processors include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive executable computer code and data from memory, e.g., a read-only memory and/or a random access memory and/or other hardware storage devices. Generally, a computer will include one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Hardware storage devices suitable for tangibly storing computer program executable computer code and data include all forms of volatile memory, e.g., semiconductor random access memory (RAM), all forms of non-volatile memory including, by way of example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD ROM disks. Any of the foregoing can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits).

A number of embodiments of the invention have been described. The embodiments can be put to various uses, such as educational, job performance enhancement, e.g., sales force and so forth. Nevertheless, it will be understood that various modifications may be made without departing from the scope of the invention.