Techniques for interactively coupling electronic content with printed media

Description

BACKGROUND OF THE INVENTION

A number of techniques for interacting with content printed on encoded media have been developed. In one technique, content is printed or written on paper containing an electro-optically readable coding pattern. The coding pattern is utilized to determine position information which can be utilized to interact with the content thereon. However, there is a substantial amount of content that has not been disseminated on non-encoded paper. Without being able to determine position information, it is difficult to deliver interactive electronic content related to such printed or written content. It is also undesirable to reproduce the substantial volume of existing literary work on encoded paper. Therefore, what is needed is a technique for delivering electronic content in an interactive session with related content printed on non-encoded media.

SUMMARY OF THE INVENTION

Embodiments of the present invention are directed toward systems and methods for interactively coupling electronic content with printed or written content. In one embodiment, the techniques include receiving an identifier of a particular written or printed content utilizing a control panel that is printed on encoded media. Electronic content related to the written or printed content may then be interactively delivered.

In another embodiment, a pre-printed control panel includes a plurality of graphics, wherein each graphics overlays a unique coded pattern. Each coded pattern is associated with a particular function or datum. A given graphic may represent a button, key, alphabetic character, numeric character, symbol, figure or the like for invoking the function or input the datum associated with the coded pattern that it overlays.

In yet another embodiment, a computing device includes an optical tracking interface adapted to receive one or more encoded patterns from a printed control panel. A processor of the computing device determines an identifier of a written or printed content from the received encoded patterns. The processor interactively delivers electronic content related to the identified written or printed content.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention are illustrated by way of example and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

FIG. 1 shows a block diagram of a computing device upon which embodiments of the present invention can be implemented.

FIG. 2 shows an exemplary media provided with a coding pattern upon which embodiments of the present invention can be implemented.

FIG. 3 illustrates an exemplary use of a control panel printed on encoded media that enables interaction between a computing device and conventional media, in accordance with one embodiment of the present invention.

FIG. 4 illustrates a first sub-control panel of an exemplary control panel, in accordance with one embodiment of the present invention.

FIG. 5 illustrates a second and third sub-control panel of an exemplary control panel, in accordance with one embodiment of the present invention.

FIG. 6 a flow diagram of a method of using a control panel printed on one encoded media to interactively couple electronic media with printed media, in accordance with one embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

Reference will now be made in detail to the embodiments of the invention, examples of which are illustrated in the accompanying drawings. While the invention will be described in conjunction with these embodiments, it will be understood that they are not intended to limit the invention to these embodiments. On the contrary, the invention is intended to cover alternatives, modifications and equivalents, which may be included within the scope of the invention as defined by the appended claims. Furthermore, in the following detailed description of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it is understood that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present invention.

FIG. 1 shows a block diagram of a computing device 100 upon which embodiments of the invention can be implemented. In general, computing device 100 may have a form factor similar to a pen, stylus or the like. The computing device 100 includes system memory 105, a processor 110, an input/output interface 115, an optical tracking interface 120, one or more buses 125, and a pointing and/or writing instrument 130. The system memory 105, processor 110, input/output interface 115 and optical tracking interface 120 are communicatively coupled to each other by the one or more buses 125.

The memory 105 may include one or more well known computer-readable media, such as static or dynamic read only memory (ROM), random access memory (RAM), flash memory, magnetic disk, optical disk and/or the like. The memory 105 may be used to store one or more sets of instructions and data that, when executed by the processor 110, cause the computing device 100 to perform the techniques for interactively coupling electronic media with printed or written media as described herein.

The computing device 100 may further include an external memory controller 135 for removably coupling an external memory 140 to the one or more busses 125. The external memory may be used to store various electronic content for use by the computing device 100. The computing device 100 may also include one or more communication ports 145 communicatively coupled to the one or more busses 125. The one or more communication ports can be used to communicatively couple the computing device 100 to one or more other devices 150. The computing device 110 may be communicatively coupled to other devices 150 by a wired communication link and/or a wireless communication link 155. Furthermore, the communication link may be a point-to-point connection and/or a network connection. The communications ports may be used to receive various electronic content from the other devices 150.

The input/output interface 115 may include one or more electro-mechanical switches operable to receive commands and/or data from a user. The input/output interface 115 may also include one or more audio devices, such as a speaker, a microphone, and/or one or more audio jacks for removably coupling an earphone, headphone, external speaker and/or external microphone. The audio device is operable to output audio content and information and/or receiving audio content, information and/or instructions from a user. The input/output interface 115 may optionally include video devices, such as an LCD display for displaying alphanumeric and/or graphical information and/or a touch screen display for displaying and/or receiving alphanumeric and/or graphical information.

The optical tracking interface 120 includes a light source or optical emitter and a light sensor or optical detector. The optical emitter may be a light emitting diode (LED) and the optical detector may be a charge coupled device (CCD) or complementary metal-oxide semiconductor (CMOS) imager array, for example. The optical emitter illuminates a surface of a media or a portion thereof and light reflected from the surface is received at the optical detector.

The surface of the media may contain a pattern detectable by the optical detector. Referring now to FIG. 2, an exemplary media 210 provided with a coding pattern, upon which embodiments of the present invention can be implemented, is shown. In one implementation, the media 210 is provided with a coding pattern in the form of electro-optically readable position code that consists of a pattern of dots 220. The dots 220 may not be easily discernible by the human visual system, and may appear as grayscale on the sheet of paper. In one implementation, the pattern comprises a unique pattern of dots 220 printed on paper or other such media. The dots 220, for example, may have a nominal spacing of 0.3 millimeters, with 669,845,157,114,773,458,169 dots encompassing an area in excess of 4.6 million square kilometers, which corresponds to about 73 trillion letter-size pages. The “encoded paper” may be subdivided into unique sections. Different sections can be used (e.g., licensed) by different entities.

As the pointing and/or writing instrument 130 and the optical tracking interface 120 move together relative to the surface, successive images are captured. The optical detector, for example, may take snapshots of the surface 100 times a second or more. By analyzing the images, position on the surface and movement relative to the surface of the media can be tracked. In one implementation, the optical detector fits the dots to a reference system in the form of a raster with raster lines 230, 240 that intersect at raster points 250. Each of the dots 220 is associated with a raster point. For example, the dot 220 is associated with raster point 250. For the dots 220 in an image, the displacement of a dot 220 from the raster point 250 associated with the dot 220 is determined. Using these displacements, the pattern in the image is compared to patterns in the reference system. Each pattern in the reference system is associated with a particular location on the surface. Thus, by matching the pattern in the image with a pattern in the reference system, the position of the writing instrument 130 relative to the surface can be determined.

By interpreting the positions of the dots 220 captured in each snapshot, the operating system and/or one or more applications executing on the computing device 100 can precisely determine the position (e.g., position in a unique region) of the computing device 100 in two dimensions. That is, for example, an x-coordinate and a y-coordinate corresponding to the position of the pointing and/or writing instrument 130 relative to the page (e.g., surface) can be determined. Furthermore, as the pointing and/or writing instrument 130 and the optical detector move together relative to the surface, the direction and distance of each movement can be determined from successive position data.

In addition, different regions of the pattern can be assigned different function, and software programs and applications may assign functionality to the various patterns of dots within a respective region. Therefore, by placing the optical detector in a particular position on the surface and performing some type of actuating event, a specific instruction, command, function, data or the like associated with the position can be entered and/or executed. For example, the pointing and/or writing instrument 130 may be mechanically coupled to an electro-mechanical switch of the input/output interface 115. Therefore, double tapping substantially the same position is detected by the electro-mechanical switch in combination with the position determined by the optical detector as an input that can cause data or a function associated with the particular position to be received or executed, respectively.

The writing element 130 can be, for example, a pen, pencil, stylus, marker or the like, and may or may not be retractable. In one or more instances, a user can use the writing element 130 to make strokes on the surface, including letters, numbers, symbols, figures and the like. These user-produced strokes can be captured (e.g., imaged and/or tracked) and interpreted by the computing device 100 according to their position on the surface on the encoded media. The position of the strokes can be determined using the pattern of dots on the surface. The strokes can also be interpreted by the computing device 100 using optical character recognition (OCR), handwriting recognition and/or handwriting understanding techniques that recognize handwritten characters (or ‘written signals’).

A user, in one implementation, uses the writing element 130 to create a character “M” at a given position on the encoded media. The user may or may not create the character in response to a prompt from the computing device 100. In one implementation, when the user creates the character, device 100 records the pattern of dots that are uniquely present at the position where the character is created. The computing device 100 associates the pattern of dots with the character just captured. When computing device 100 is subsequently positioned over the “M,” the computing device 100 recognizes the particular pattern of dots associated therewith and recognizes the position as being associated with an “M.” In effect, the computing device 100 recognized the presence of the character using the pattern of markings at the position where the character is located, rather then by recognizing the character itself.

In another implementation, the computing device 100 analyzes the pattern of dots that are uniquely present at the position where the character is created (e.g., stroke data). Using a character recognition application, the stroke data captured by analyzing the pattern of dots can be read and translated into the character “M.”

In another implementation, a character is associated with a particular command. For example, a user can write a character composed of a circled “M” that identifies a particular command, and can invoke that command repeatedly by simply positioning the optical detector over the written character. In other words, the user does not have to write the character for a command each time the command is to be invoked; instead, the user can write the character for a command one time and invoke the command repeatedly using the same written character.

In another implementation, the encoded paper may be preprinted with one or more graphics at various locations in the pattern of dots. For example, the graphic may be a preprinted graphical representation of a button. The graphics lies over a pattern of dots that is unique to the position of the graphic. By placing the optical detector over the graphic, the pattern of dots underlying the graphics are read (e.g., scanned) and interpreted, and a command, instruction, function or the like associated with that pattern of dots is implemented by the computing device 100 or data associated with that pattern is received as an input by the computing device 100. Furthermore, some sort of actuating event or movement may be performed using the computing device 100 in order to indicate that the user intends to invoke the command, instruction, function or the like associated with the graphic and/or receive the data as an input.

In yet another implementation, the optical tracking interface 120 may be used to read a barcode. By analyzing the pattern of the barcode, the optical detector in combination with the processor 110, executing an applicable application, can determine one or more data and/or one or more commands associated with the particular barcode. For example, the bar code on the back of the book may be decoded to determine what book it is.

Referring now to FIG. 3, an exemplary use of a control panel printed on encoded media 305 that enables interaction between a computing device 310 and conventional media 315, in accordance with one embodiment of the present invention, is illustrated. The conventional media 315 may include printed or written content, including text and graphics, utilized to disseminate literary work.

The control panel printed on encoded media 305 may include one or more graphics at various regions in the coding pattern. The pattern of dots underlying each graphic may be associated with a given datum or function. The graphics may include graphical representations of one or more bottoms, one or more icons, one or more keys, one or more alphabetic characters, one or more numeric characters, one or more designs, one or more symbols, one or more figures and/or the like. For example, the control panel 305 may include graphical representations of “start,” “enter,” and “repeat” buttons 330. The underlying pattern of dots corresponding to the particular graphical representation of the “enter” button is pre-associated with an input for invoking an enter function in one or more application running on the pen-shaped computing device. Accordingly, when a user taps on the “enter” button an enter function is performed. The control panel 305 may also, for example, include a plurality of alphabetic characters 325. The underlying pattern of dots proximate a given alphabetic character is pre-associated with the particular letter. Accordingly, when a user taps on the “J” graphic, a computing device-readable representation of the letter J is received as an input.

It is appreciated that the illustrated control panel on encoded media 305 is for illustration only and is not intended to limit the scope of the invention. Instead, the control panel may include graphical representation of any number and/or combination of pre-associated functions, data, and open input regions. Furthermore, the pattern of dots visible in the exemplary control panel 305 is for illustrative purposes. Normally, the dot pattern would not be easily discernable to a user.

The control panel 305 may be produced on encoded media, such as paper, cardboard, plastic or the like. The coding pattern on the media is electro-optically readable by the computing device 310. The control panel may have a form factor of a card, sheet, book mark, poster, folder, pamphlet, leaflet, folio, journal or the like. The form fact of the control panel may have dimensions the same as or similar to a business card, postcard, book marker, standard page size, book cover. In one implementation, a first set of controls (e.g., graphics) may be printed on a first surface of the control panel 305 and a second set of controls (e.g., graphics) may be printed on a second surface. In another implementation, the controls may be printed in a first orientation (e.g., portrait) on a first surface and the same controls may be printed in a second orientation (e.g., landscape) on a second surface.

The user may hold the computing device 310 in a similar manner to holding a pen, and therefore it may be referred to as a pen-shaped computing device 310. In particular, as the user points, taps, writes or the like with the pen-shaped computing device 310, the Operating System (OS) or another application collects position information associated with the user's actions on the encoded control panel. The optical tracking interface 120 of the pen-shaped computing device 310, for example, may take snapshots of the surface of the encoded control panel 305 approximately 10-1000 times per second. By analyzing the images of the position code, movement relative to the surface can be tracked. The relative movement can be used to capture the selection of functions and/or input of data that are pre-associated with particular regions of the encoded pattern, and/or stroke data (e.g., writing) in open input regions.

In one example, a student may enter the international standard book number (ISBN) of a work book by tapping on the corresponding alphanumeric keys 325, 330 printed on the control panel 305. The student may then tap enter 335 on the control panel 305 using the pen-shaped computing device 310. The student may enter the international standard book number (ISBN) in response to an audio prompt 340. The prompt 340 may be generated when a particular application is launched on the pen-shaped computing device 310 by the student. The pen-shaped computing device 310 utilizes the ISBN to determine which book is being used by the student. The student may also enter the page number 345 and then tap enter 335 to begin at a given page within the book. The student may then receive interactive audio content corresponding to the given page in the book from the pen-shaped computing device 310. For example, the portion of the content corresponding to the indicated page could be output by the pen-shaped computing device 310 as an audio file 340.

In another implementation, the student may enter the tile of the book using the graphical representation of the alphabetic characters 3325, instead of the ISBN. In yet another implementation, the student may use the optical tracking interface 120 to scan a barcode on the book. An application executing on the processor 110 may determine the identity of the particular book from the barcode.

Another exemplary control panel is illustrated in FIGS. 4 and 5. The illustrated control panel 410 may be in the shape of a pamphlet. A first sub-control panel is printed on the first surface (e.g., front) of the pamphlet, as depicted in FIG. 4. A second sub-control panel is printed on the left inside panel and a third sub-control panel is printed on the right inside panel of the pamphlet, as depicted in FIG. 5. The first sub-control panel may include common functions represented by graphical buttons, such as start, enter and repeat buttons. The first sub-control panel may also include general alphabetic and numeric data entry buttons. The first sub-control panel may also include additional graphical regions for invoking other functions and data, such as activity selection buttons 415 and the like. The other sub-control panels may include one or more of the same graphical buttons that are on the first sub-control panel and also additional graphics for invoking other functions and/or entering other data. Any of the sub-panels may also include open input regions 510 for capturing hand written inputs from the user.

A user, such as a student, may use the control panel 410 to interactively couple electronic content with printed or written content. In particular, the student may receive audio content 340 from the pen-shaped computing device 310 related to the content in a given book 315. In one example, the student may take a quiz about the subject matter on a given page of the book using the pen-shaped computing device. The quiz question may be generated based upon the content of the book or may be those present in the book. For instance, the student may enter the page number 345 and question number 350 of a question 355 appearing in the book 315. The question may be output as an audio prompt and the student may use one or more graphical buttons, such as true and false buttons, multiple choice buttons or the like, to answer the question. The questions may be framed within an engaging theme, such as a quick quiz game, a word storm game, a memory match game or the like. The pen-shaped computing device 310 may also provide audio scaffolded hints for answering the questions, an indication of whether the answer provided by the student via the control panel 410 is correct or not, and other pedagogy.

In yet another example, a language button may be provided on the control panel. Selection of the language button may then provide interactive content in the native language of the user in accordance with a language selected during a configuration of the pen-shaped computing device or in response to an additional prompt when the language button is taped by the student. In addition, the control panel 410 may include audio control graphical buttons, such as up volume, down volume, mute and the like, to adjusting the audio output.

Referring now to FIG. 6, a method of using a control panel printed on encoded media to interactively couple electronic content with printed or written media, in accordance with one embodiment of the present invention is shown. The method includes receiving an identifier of written or printed content utilizing a control panel printed on encoded media, at 610. In one implementation, a user may input a title of the written or printed content by performing actuating events relative to one or more graphics overlaying particular position coded regions each of which is pre-associated with a particular function or datum. For instance, a user may tap on one or more graphical representations of alphanumeric keys to enter the title of a particular book. In another implementation, the user may input the ISBN of the book. In yet another implementation, the user may input a barcode identifying the book. In other implementations, the user may identify the content any number of other ways.

Optionally, the method may also include receiving one or more identifiers of a particular portion of the written or printed content, at 620. For instance, the user may enter a particular page number of the book, a range of page numbers, a chapter number, section identifier, a subsection identifier, a heading, a volume number, an issue number or the like, by tapping on the appropriate graphical representations of alphanumeric keys.

At 630, interactive electronic content related to the written or printed content is delivered. Furthermore, if a identifier of a particular portion of the written or printed content is received, a particular portion of the electronic content related to the particular portion of the written or printed content may be selectively delivered. In addition, if one or more identifiers of additional portions of the written or printed content are received, the additional portions of the electronic content related to the identified portions of the written or printed content may be delivered in response thereto.

The interactive electronic content may provide directed user activities and/or group activities. For example, the interactive content may include one or more contextually-relevant teaching points and sound effect that engage a student in the content of the conventional printed or written media. The teaching points may include factoids, quick reading comprehension suggestions and/or the like. Sound effects may be utilized to put the reader “in the world” that they are reading about and provide additional “entry points” for neural net recall.

The interactive content may also include touch diagram and other visual aids to provide help with strategies that may be challenging when a user is first asked to go it alone. The visual aids may include Venn diagrams, sequence charts, main idea/details, thinking aloud, cause-effect charts and/or the like. Use of the strategies are directly related to the content or a particular portion of the printed or written media.

The interactive content may also include engaging interactions such as game models, such as word storm, quick quiz, memory match and/or the like. For example, in word storm two or more users are given a category. Each user is given a period to generate a list of words that relate to some aspect of the content of the written or printed media. The user may write their list on encoded paper or plain paper. After the period ends the users compare lists and score based on the correctness, uniqueness and/or the like of their responses. In quick quiz, one or more users may answer true/false, multiple choice, free-entry 1-2 word answers in reading comprehension quizzes. In memory match a user may write audio hints in individual regions and then match pairs of audio clues. Such activities may encourage careful listening, group discussion, resolution of conflicts and ambiguities, compare and contrast analysis, creation of visual mnemonics and/or the like.

In one implementation, the interactive content provided by the computing device may be stored in the memory 105. In another implementation, the interactive content provided by the computing device may be received via an external memory 140 such as a memory cartridge that can be removable coupled to the computing device. In yet another implementation, the interactive content may be received via a communication link with another computing device 150.

Optionally, the method may also include reporting parameters concerning the interactive session, at 640. In particular, logged information collected by the computing device 100 during one or more interactive sessions may be reported.

Embodiments advantageous enable a user to interact with written and printed media using a computing device. In particular, the control panel for non-encoded content can be used to enable interaction between a pen-shaped computing device and printed books, pamphlets and the like. Thus, content on conventional written and printed media is no longer static. Instead, the usefulness of content on non-encoded media is increased by use of the control panel printed on encoded media. The control penal also benefits publishers and users by reducing or eliminating the cost of re-publishing content on encoded media.

The foregoing descriptions of specific embodiments of the present invention have been presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and obviously many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, to thereby enable others skilled in the art to best utilize the invention and various embodiments with various modifications as are suited to the particular use contemplated. It is intended that the scope of the invention be defined by the Claims appended hereto and their equivalents.