SYSTEM FOR COSMETIC APPLICATION SKILL IMPROVEMENT

Description

FIELD OF THE INVENTION

The present invention relates generally to the field of cosmetics and, more specifically, to a system that may provide users with an overlay projection guide to assist in improving the user's skills in applying makeup and other cosmetic products, using machine learning, artificial intelligence, augmented reality, virtual reality, and other technologies.

BACKGROUND

The background description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description that may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present disclosure.

Applying makeup can be a challenging task for many individuals, and conventionally requires practice and skill development to achieve desired results. The process of applying makeup often involves trial and error and can be time-consuming and frustrating without proper guidance and tracking of the user's progress and improvement. Traditionally, cosmetic application skills are learned manually, relying heavily on hands-on practice and observational learning. This approach has limitations such as a lack of personalized instruction, difficulty in tracking progress, and limited access to expert guidance.

SUMMARY

The present invention provides a system that may provide users with an overlay projection guide to assist in improving the user's skills in applying cosmetic products, using technologies including machine learning, artificial intelligence, virtual reality, and augmented reality.

In one aspect, a system may comprise one or more sensors configured to capture, in real-time, one or more images of a face of a user. The system may also include one or more processors and one or more memories storing non-transitory computer-readable instructions that, when executed by the one or more processors, cause the one or more processors to capture, using the one or more sensors, the one or more images of the face of the user. The computer-readable instructions, when executed by the one or more processors, may also cause the one or more processors to analyze the one or more images of the face of the user as the user applies one or more cosmetic products to at least one facial feature of the user to identify techniques used by the user. The computer-readable instructions, when executed by the one or more processors, may also cause the one or more processors to evaluate the techniques used by the user using a precision detection machine learning model. The precision detection machine learning model may be trained using training data of previous cosmetic applications. The precision detection machine learning model may also determine a level of precision associated with the techniques used by the user.

Additionally, the computer-readable instructions, when executed by the one or more processors, may cause the one or more processors to generate in real-time, based on the determined level of precision determined by the precision detection machine learning model, an overlay projection of cosmetic product application guidance associated with the one or more images of the face of the user. The computer-readable instructions, when executed by the one or more processors, may then cause the one or more processors to, using an augmented reality module, display the overlay projection of cosmetic product application guidance onto one of: the one or more images of the face of the user, a reflection of the face of the user as it appears in a mirror in real-time, or the face of the user in real-time. The system may include additional, less, or alternate elements, including those discussed elsewhere herein.

In another aspect, the system may comprise one or more sensors configured to capture, in real-time, one or more images of a face of a user, one or more processors, and one or more memories storing non-transitory computer-readable instructions that, when executed by the one or more processors, may cause the one or more processors to receive an image of a face depicting a user's desired makeup look. The computer-readable instructions, when executed by the one or more processors, may also cause the one or more processors to capture, using the one or more sensors, the one or more images of the face of the user and compare the image of the face depicting the user's desired makeup look to the one or more images of the face of the user, in order to identify techniques used by the user. The computer-readable instructions, when executed by the one or more processors, may also cause the one or more processors to evaluate the techniques used by the user using a precision detection machine learning model. The precision detection machine learning model may be trained using training data of previous cosmetic applications. The precision detection machine learning model may also determine a level of precision associated with the techniques used by the user.

Additionally, the computer-readable instructions, when executed by the one or more processors, may cause the one or more processors to generate in real-time, based on the determined level of precision determined by the precision detection machine learning model, an overlay projection of cosmetic product application guidance for achieving the user's desired makeup look. The computer-readable instructions, when executed by the one or more processors, may then cause the one or more processors to use an augmented reality module to display the overlay projection of cosmetic product application guidance onto one of: the one or more images of the face of the user, a reflection of the face of the user as it appears in a mirror in real-time, or the face of the user in real-time. The system may include additional, fewer, or alternate elements, including those discussed elsewhere herein.

In yet another aspect, a method and technique for operating a system may include capturing, using one or more sensors, one or more images of a face of a user. The method may also include analyzing, by one or more processors, the one or more images of the face of the user as the user applies one or more cosmetic products to at least one facial feature of the user to identify techniques used by the user. The method may include evaluating by the one or more processors, the techniques used by the user using a precision detection machine learning model. The precision detection machine learning model may be trained using training data of previous cosmetic applications. The precision detection machine learning model may also determine a level of precision associated with the techniques used by the user.

Additionally, the method may include generating by the one or more processors, in real-time, based on the determined level of precision determined by the precision detection machine learning model, an overlay projection of cosmetic product application guidance associated with the one or more images of the face of the user. The method may also include using an augmented reality module, displaying, by the one or more processors, the overlay projection of cosmetic product application guidance onto one of: the one or more images of the face of the user, a reflection of the face of the user as it appears in a mirror in real-time, or the face of the user in real-time. The method may include additional, fewer, or alternative actions, including that discussed elsewhere herein.

Advantages will become more apparent to those of ordinary skill in the art from the following description of the preferred embodiments which have been shown and described by way of illustration. As will be realized, the present embodiments may be capable of other and different embodiments, and their details are capable of modification in various respects. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.

BRIEF DESCRIPTION OF THE DRAWINGS

The figures described below may depict various aspects of the system and methods disclosed herein. It should be understood that each figure depicts an embodiment of a particular aspect of the disclosed system and methods, and that each of the figures is intended to accord with a possible embodiment thereof.

There are shown in the drawings arrangements which are presently discussed, it being understood, however, that the present embodiments are not limited to the precise arrangements and instrumentalities shown, wherein:

FIG. 1 depicts an exemplary computer system for cosmetic skill improvement, according to some embodiments;

FIG. 2 depicts a user's initial application of a cosmetic product to a user's face, according to some embodiments;

FIG. 3 depicts an exemplary handheld device displaying an indication of a user's level of precision in applying a cosmetic product prior to receiving cosmetic product application guidance, according to some embodiments;

FIG. 4 depicts an exemplary handheld device displaying an overlay projection of cosmetic product application guidance over an image of a user's face, according to some embodiments;

FIG. 5 depicts an exemplary mirror upon which an overlay projection of cosmetic product application guidance is displayed over a reflection of a user's face, according to some embodiments;

FIG. 6 depicts an exemplary user applying a cosmetic product to the user's face based on cosmetic product application guidance, according to some embodiments;

FIG. 7 depicts a user's final look after applying a cosmetic product to the user's face based on cosmetic product application guidance, according to some embodiments;

FIG. 8 depicts an exemplary handheld device displaying an indication of a user's new level of precision in applying a cosmetic product after receiving cosmetic product application guidance, according to some embodiments;

FIG. 9 depicts an example flow diagram of an exemplary method for generating and providing cosmetic product application guidance in real-time as a user applies cosmetic products, according to some embodiments; and

FIG. 10 depicts example flow diagram of an exemplary method for generating and providing cosmetic product application guidance for a user based on a user's desired makeup look, according to some embodiments.

While the systems and methods disclosed herein are susceptible of being embodied in many different forms, they are shown in the drawings and are described herein in detail specific exemplary embodiments thereof, with the understanding that the present disclosure is to be considered as an exemplification of the principles of the systems and methods disclosed herein and is not intended to limit the systems and methods disclosed herein to the specific embodiments illustrated. In this respect, before explaining at least one embodiment consistent with the present systems and methods disclosed herein in detail, it is to be understood that the systems and methods disclosed herein are not limited in its application to the details of construction and to the arrangements of components set forth above and below, illustrated in the drawings, or as described in the examples.

Methods and apparatuses consistent with the systems and methods disclosed herein are capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein, as well as the abstract included below, are for the purposes of description and should not be regarded as limiting.

DETAILED DESCRIPTION

Overview

The present disclosure provides a system and techniques that may utilize smart technologies to superimpose a guided makeup look onto a user's face, thus delivering a personalized and precise makeup application experience. The system may capture images of the face of the user and in some embodiments, allow the user to upload a desired makeup look. The system may analyze the images as the user applies the makeup. The system may then identify techniques used by the user and evaluate the techniques using a precision detection machine learning model. The model may determine a precision level and the processor may generate an overlay projection which may provide guidance to the user in applying the makeup. The overlay projection may provide step-by-step instructions and visual cues to help the user apply the makeup accurately and efficiently.

The system may use machine learning (ML), artificial intelligence (AI), augmented reality (AR), virtual reality (VR), and other technologies to assist the user in accurately applying the makeup. This may include re-creating looks selected by the user.

These different forms of technology may personalize user experiences and offer assistance in makeup application improvement. Additionally, the system may offer integration with smart devices and smart packaging, allowing the system to create and use a wide variety of makeup looks and styles. The system may also include a feature for users to subscribe to, download, and share makeup looks on social media platforms. As provided herein, the system may streamline the cosmetic application process, providing users with a personalized, efficient, and precise makeup experience.

The system provided herein may embody a unique blend of technology and usability to enhance the user's makeup application technique based on a user's level of precision. Advantageously, the system provided herein may incorporate features such as analyzing a user's images, face, or reflection in a mirror while applying makeup, evaluating the user's techniques, determining a precision level, generating an overlay based on the precision level, and displaying a projection of the overlay for user guidance, in various embodiments. The integration of AR for detailed facial mapping and makeup preview may further distinguish this system, providing users an immersive and interactive makeup experience. As an added benefit, the system may continue to learn and adapt based on user preferences and techniques, software updates, and updated makeup looks to capture the latest trends.

The system may include one or more sensors, processors, and memories. Using the system and its sensors, the user may capture high-resolution images and/or video of their face before and after applying makeup using a camera, smartphone, mirror, or any other suitable image-capturing device. The system may also incorporate specific guidelines for image capture, such as lighting conditions, angles, and distances, to ensure consistency and accuracy in the evaluation process. The system may use different forms of technology, such as machine learning algorithms and artificial intelligence to evaluate the user's techniques when applying the makeup. These forms of technology may also learn the user's facial features, skin type, and preferences over time. The precision detection machine learning model may be trained using previous makeup applications and may determine a level of precision based on the user's techniques.

The machine learning algorithms may be trained and updated over time to improve their accuracy and functionality. The AR software may enable the user to see a virtual overlay projection of the makeup application prior to applying the makeup. The overlay projection may be designed to guide the user's makeup application by providing feedback superimposed over the user's face in images of the face of the user, the reflection of the face of the user as it appears in a mirror, or the face of the user in real-time.

In other examples, the system may allow the user to upload an image of a face depicting the user's desired makeup look. The system may compare this desired makeup look to the images of the face of the user, which were captured by the system's sensors. This comparison may assist in identifying techniques used by the user. The techniques may then be evaluated using a machine learning model which may be trained using data from previous makeup applications. The model may also determine a level of precision based on the user's techniques and generate an overlay projection providing guidance for the user to follow. This overlay projection may assist in guiding the user to the desired makeup look previously uploaded. This overlay projection may be displayed, using an augmented reality module, onto the images of the face of the user, a reflection of the face of the user in a mirror in real-time, or the face of the user in real-time.

The system may provide users with a personalized, efficient, and precise makeup experience, eliminating the need for manual adjustments and reducing the likelihood of uneven or inaccurate makeup application. The system may assist individuals in evaluating and improving their skills in applying makeup and other cosmetic products. The system may also have the potential to revolutionize the makeup application process by providing users with a personalized, efficient, and precise makeup experience. The system's innovative use of AI, ML, VR, AR, and computer vision technologies may enable it to adapt to each user's unique facial features, preferences, and skill level reducing the likelihood of uneven or inaccurate makeup application. With the addition of the features mentioned above, the system may provide even greater convenience and functionality for users, making it a valuable addition to any makeup routine.

By incorporating these advanced features and enhancements, the system may further revolutionize the makeup application process and expand its applications. The system may become an indispensable tool for users looking for a personalized, efficient, and precise makeup experience, making it an attractive option for both personal and professional use. Overall, the system may represent a significant advancement in the field of cosmetic application systems. By leveraging AI, ML, and computer vision technologies, the system may provide users with a personalized, efficient, and precise makeup experience that adapts to each user's unique facial features, preferences, and skill level. The system may streamline the makeup application process, ensuring improved results with reduced effort.

The integration of additional features such as VR and AR technologies and adaptive makeup tutorials, may further enhance the user experience and expand the system's applications. The system may have the potential to revolutionize makeup application and skill improvement in makeup application for users whose skill levels range from beginner to expert. By simplifying the makeup application process and assisting the user in improving their skills, the system may not only save users time and effort but also boost their skill, confidence, and self-esteem.

Example System

FIG. 1 depicts an exemplary computer system for cosmetic skill improvement, according to some embodiments. The high-level architecture illustrated in FIG. 1 may include both hardware and software components, as well as various data communications channels for communicating data between the various hardware and software components, as is described below.

The system 100 may include one or more server(s) 110, one or more user devices 120 (which may include, e.g., smart phones, smart watches or fitness tracker devices, tablets, laptops, virtual reality headsets, smart or augmented reality glasses, wearables, etc.), and/or a smart mirror 140. The server(s) 110, user device(s) 120, and/or smart mirror 140 may be configured to communicate with one another via a wired or wireless computer network 130, and/or via short range signals, such as BLUETOOTH signals.

Although one server 110, one user device 120, one smart mirror 140, and one network 130 are shown in FIG. 1, any number of such servers 110, user devices 120, smart mirror 140, and networks 130 may be included in various embodiments. To facilitate such communications, the servers 110, user devices 120, and/or smart mirror 140 may each respectively comprise a wireless transceiver to receive and transmit wireless communications.

The one or more servers 110 of the system 100 may include one or more processors 112 and/or one or more memories 114. In some embodiments the server 110 may comprise one or more servers, which may comprise multiple, redundant, or replicated servers as part of a server farm. In still further aspects, such server(s) 110 may be implemented as cloud-based servers, such as a cloud-based computing platform. For example, such server(s) 110 may be any one or more cloud-based platform(s) such as MICROSOFT AZURE, AMAZON AWS, or the like. Such server(s) 110 may include one or more processor(s) 112 (e.g., CPUs) as well as one or more computer memories 114.

The one or more memories 114 of the one or more servers 110 may store one or more machine learning models 116 and/or one or more machine learning model training applications 118. The memories 114 of the one or more servers 110 may include one or more forms of volatile and/or non-volatile, fixed and/or removable memory, such as read-only memory (ROM), electronic programmable read-only memory (EPROM), random access memory (RAM), erasable electronic programmable read-only memory (EEPROM), and/or other hard drives, flash memory, MicroSD cards, and others. Memories 114 may store an operating system (OS) (e.g., Microsoft Windows, Linux, UNIX, etc.) capable of facilitating the functionalities, apps, methods, or other software as discussed herein. The stored machine learning models 116 may include, for instance, a machine learning model trained to analyze images associated with makeup looks to identify a user's techniques used to create the makeup looks, a machine learning model trained to analyze data associated with the user's techniques in order to determine a level of precision, a machine learning model trained to identify the techniques used by the user, etc.

Additionally, or alternatively, the memories 114 may store makeup look data, previous cosmetic application data, and/or user data. The makeup look data may include, for instance, the user's or other user's previous makeup application looks. Makeup look data may also include guidance, tutorials, etc., associated with various makeup looks, and may also be stored in a look database 132 (or in multiple such databases), which may be accessible or otherwise communicatively coupled to the server 110. The previous cosmetic application data may include image and/or video data from a user's previous application of cosmetic products. This image and/or video data may be stored in a cosmetic application database 136 (or in multiple such databases). The user data may include previous makeup looks worn by the user, user preferences, and various other data associated with the user, and may also be stored in a user database 134 (or in multiple such databases), which may be accessible or otherwise communicatively coupled to the server 110. Furthermore, in some examples, the makeup look data, previous cosmetic application data, and the user data may be stored in the same database, which may be accessible or otherwise communicatively coupled to the server 110.

Furthermore, the memories 114 may store instructions that, when executed by the processors 112, cause the processors 112 to receive data from various databases such as the databases 132, 134, and 136 and/or data from the smart mirror 140 and/or the user device 120 (e.g., via the network 130). The data from the smart mirror 140 and/or the user device 120 may include, for instance, data captured by the sensors 142 of the smart mirror 140 and/or data captured by the sensors 122 of the user device 120, data input from a user via a user interface 143 of the smart mirror 140 and/or data input from a user via the user interface 124 of the user device 120, etc. The instructions stored on the memories 114, when executed by the processors 112, may cause the processors 112 to analyze data received from the database, and/or the smart mirror 140 and/or the user device 120 in order to make an identification or a prediction (e.g., an identification of a technique used by a user as the user applies a cosmetic product, an identification of a technique used by another user in applying a cosmetic product shown in an image, an identification of a precision level associated with a user's application of a cosmetic product, a prediction of one or more techniques and/or products that may be used by a user to replicate a desired makeup look, etc.) based on the received data, and subsequently send the identification and/or prediction to the smart mirror 140 and/or the user device 120. For instance, this analysis and identification and/or prediction may be based upon applying a trained machine learning model 116 to the data received from the databases 132, 134, and 136 and/or the smart mirror 140 and/or the user device 120.

In some examples, one or more machine learning model(s) 116 may be executed on the server 110, while in other examples one or more machine learning model(s) 116 may be executed on another computing system, separate from the server 110. For instance, the server 110 may send data to another computing system, where a trained machine learning model 116 is applied to the data, and the other computing system may send a prediction or identification, based upon applying the trained machine learning model 116 to the data, to the server 110. Moreover, in some examples, one or more machine learning model(s) 116 may be trained by respective machine learning model training application(s) 118 executing on the server 110, while in other examples, one or more machine learning model(s) 116 may be trained by respective machine learning model training application(s) 118 executing on another computing system, separate from the server 110.

Whether the machine learning model(s) 116 are trained on the server 110 or elsewhere, the machine learning model(s) 116 may be trained by respective machine learning model training application(s) 118 using training data (including historical data in some cases), and the trained machine learning model(s) 116 may then be applied to new/current data that is separate from the training data in order to determine, e.g., predictions and/or identifications related to the new/current data.

For instance, a machine learning model trained to analyze data associated with a user's application of a cosmetic product to identify one or more techniques used by the user in applying the cosmetic product may be trained by a machine learning training application 118 using training data including images and/or videos of various users applying cosmetic products using various techniques that are labeled with techniques used in each of the images and/or videos.

Furthermore, a machine learning model trained to analyze images of finished makeup looks to identify one or more techniques used to apply the finished makeup looks may be trained by a machine learning training application 118 using training data including images of various users wearing completed makeup looks, that are labeled with techniques used to apply each of the completed makeup looks.

Moreover, a machine learning model trained to analyze data associated with a user's application of a cosmetic product, or images of finished makeup looks, to identify a level of precision used by a user applying the cosmetic product and/or makeup look, may be trained by a machine learning training application 118 using training data including images and/or videos of various users applying cosmetic products that are labeled with levels of precision used by the user, and/or images of various users wearing completed makeup looks, that are labeled with levels of precision used to apply the completed makeup looks.

For example, a machine learning model 116 trained to analyze data associated with a user's face to identify facial features thereon may be trained by a machine learning model training application 118 using training data including images of various faces and/or three-dimensional maps or overlay projections associated with the various faces, and indications of locations of facial features in the images and/or three-dimensional maps or overlay projections.

For instance, each image and/or three-dimensional map or overlay projection may be labeled to indicate locations of facial features such as the eyes, eyelids, eyebrows, eyelashes, cheeks, cheekbones, nose, lips, chin, etc. on the face, and these labeled images and/or three-dimensional maps or overlay projections may be used as training data. Once sufficiently trained using this training data, such a machine learning model 116 may be applied to a new image, video, and/or three-dimensional map associated with a user's face (e.g., an image or video captured by the sensors 142, 124, etc., in real-time), and may identify likely locations of various facial features of the user's face. Additionally, training data of previous cosmetic applications may include images of users performing historically identified techniques and historical levels of precision associated with each historically identified technique.

As another example, a machine learning model 116 trained to analyze images or videos to identify techniques used to create makeup looks shown in the images or videos may be trained by a machine learning model training application 118 using training data including images of individuals with various makeup looks applied, and indications of the techniques that were used to create the looks shown in the images. The machine learning model 116 may be trained by a machine learning model training application 118 to determine a precision level.

For instance, an image of an individual wearing a particular makeup look may be labeled as a particular look, with the particular look including techniques such as symmetry and balance, blending and color matching, precision of lines and edges, or overall aesthetic appeal used to create that particular look, etc. These labeled images may be used as training data for a similar look. Once sufficiently trained using this training data, such a machine learning model 116 may be applied to a new image, reflection, or video such as an image provided from a user via a user interface 143 and/or a user interface 124, or an image from a social media link provided by the user via the user interface 143 and/or a user interface 124, and may identify/predict techniques that may be used to replicate the makeup look shown in the image, reflection, or video. In some examples, the machine learning model 116 may further generate guidance, including step-by-step guidance, to be used by the smart mirror 140 when providing guidance, instructions, tutorials, feedback, etc., for replicating the makeup look shown in the image or the look desired by the user. The machine learning model 116, once applied, may provide a replicated image, reflection, or video, which may then be used to generate the overlay projection providing the necessary guidance and instructions for completion of the look.

Additionally, as another example, a machine learning model 116 trained to analyze data associated with previous makeup looks selected from a user to predict additional makeup looks for the user may be trained by a machine learning application 118 using training data including makeup looks selected by previous users, characteristics of the previous users, input/feedback from the previous users about the makeup looks, once applied by a smart mirror 140, etc. For instance, various makeup looks may be labeled with indications of characteristics of users who gave positive feedback regarding the makeup looks, indications of other looks receiving positive feedback from the same users, etc. Once sufficiently trained using this training data, such a machine learning model 116 may be applied to a user, the user's characteristics, and previous makeup looks selected/liked by the user and may predict/suggest other makeup looks that the user may enjoy and provide overlay projections based on this.

In various aspects, the machine learning model(s) 116 may comprise machine learning programs or algorithms that may be trained by and/or employ neural networks, which may include deep learning neural networks, or combined learning modules or programs that learn in one or more features or feature datasets in particular area(s) of interest. The machine learning programs or algorithms may also include natural language processing, semantic analysis, automatic reasoning, regression analysis, support vector machine (SVM) analysis, decision tree analysis, random forest analysis, K-Nearest neighbor analysis, naïve Bayes analysis, clustering, reinforcement learning, and/or other machine learning algorithms and/or techniques.

In some embodiments, the artificial intelligence and/or machine learning based algorithms used to train the machine learning model(s) 116 may comprise a library or package executed on the server 110 (or other computing devices not shown in FIG. 1). For example, such libraries may include the TENSORFLOW based library, the PYTORCH library, and/or the SCIKIT-LEARN Python library. Machine learning may involve identifying and recognizing patterns in existing data (such as training a model based upon historical data) in order to facilitate making predictions or identification for subsequent data (such as using the machine learning model on new/current data order to determine a prediction or identification related to the new/current data).

Machine learning model(s) may be created and trained based upon example data (e.g., “training data”) inputs or data (which may be termed “features” and “labels”) in order to make valid and reliable predictions for new inputs, such as testing level or production level data or inputs. In supervised machine learning, a machine learning program operating on a server, computing device, or otherwise processor(s), may be provided with example inputs (e.g., “features”) and their associated, or observed, outputs (e.g., “labels”) in order for the machine learning program or algorithm to determine or discover rules, relationships, patterns, or otherwise machine learning “models” that map such inputs (e.g., “features”) to the outputs (e.g., labels), for example, by determining and/or assigning weights or other metrics to the model across its various feature categories. Such rules, relationships, or otherwise models may then be provided subsequent inputs in order for the model, executing on the server, computing device, or otherwise processor(s), to predict, based upon the discovered rules, relationships, or model, an expected output.

In unsupervised machine learning, the server, computing device, or otherwise processor(s), may be required to find its own structure in unlabeled example inputs, where, for example multiple training iterations are executed by the server, computing device, or otherwise processor(s) to train multiple generations of models until a satisfactory model, e.g., a model that provides sufficient prediction accuracy when given test level or production level data or inputs, is generated. The disclosures herein may use one or both of such supervised or unsupervised machine learning techniques.

In addition, memories 114 may also store additional machine readable instructions, including any of one or more application(s), one or more software component(s), and/or one or more application programming interfaces (APIs), which may be implemented to facilitate or perform the features, functions, or other disclosure described herein, such as any methods, processes, elements or limitations, as illustrated, depicted, or described for the various flowcharts, illustrations, diagrams, figures, and/or other disclosure herein. For instance, in some examples, the computer-readable instructions stored on the memory 114 may include instructions for carrying out any of the steps of the method 900 and/or 1000 via an algorithm executing on the processors 112, which is described in greater detail below with respect to FIG. 9 and FIG. 10. It should be appreciated that one or more other applications may be envisioned and that are executed by the processor(s) 112. It should be appreciated that given the state of advancements of mobile computing devices, any or all of the processor's functions and steps described herein may be present together on a mobile computing device, such as the user device 120, or the smart mirror 140.

The system 100 may also include one or more user devices 120. The user device 120 may be any type of user device such as a smartphone, tablet, laptop, smart watch, desktop computer, gaming console, smart home device, virtual reality headset, and/or any other device which includes sensors for capturing images and/or video of the user's face. The user device(s) 120 may include one or more sensors 122, a user interface 124, one or more light sources 126 which may be configured to provide light to the face of the user of the user device 120, one or more processors 127, one or more memories 128, and/or an augmented reality module 129. Moreover, the user device 120 may include one or more processor(s) 127, as well as one or more computer memories 128.

The one or more sensors 122 may be configured to capture real-time data associated with the face of a user before, during, and/or after a user applies a cosmetic product to the user's face. The sensors 122 may include, for instance, a camera (in some cases, a high resolution camera) and/or a depth sensor configured to capture data associated with the user's face, data associated with various cosmetic products to be applied to the user's face and/or their packaging, etc. Moreover, in some examples, the sensors 122 may include sensors (e.g., the camera and/or the depth sensor, or additional or alternative sensors) configured to capture biometric data associated with the user, such as facial recognition data, fingerprint recognition data, iris recognition data, etc.

The user device 120 may include, or may be configured to communicate with, the user interface 124, which may receive input from users and may provide audible or visible output to users. The user interface 124 may be configured to receive inputs and selections from the user of the user device 120, and/or to provide audible or visual feedback to the user of the user interface 124, including instructions, guidance, tutorials, etc., associated with the user applying cosmetic products to the user's face for a particular makeup look selected by the user. For instance, the user interface 124 may be configured to receive an indication of a makeup look from a user and provide guidance associated with applying one or more cosmetic products to the facial features of the user in order to achieve a desired makeup look, based on real-time data captured by the sensors 122. In some embodiments, the user interface 124 may receive input from the user indicating a selection of a makeup look. The selected makeup look may be associated with various parameters and/or specifications that the user device 120 may use to provide guidance to the user so that the user can apply cosmetic products to his or her face to achieve the selected look. Additionally, the user interface 124 may receive an image or a social media link from the user.

Moreover, in some examples, the user interface 124 may be configured to provide feedback associated with a selected makeup look after the selected makeup look is applied by the user. This feedback may include an indication of a level of precision based on the user's techniques and application skills. This precision level may range from a beginner level to an expert level. Based on the user's precision level, the user interface 124 may direct the user to resources to assist the user in improving their makeup application skills. These resources may include internal resources such as photos, videos, messages, etc. stored on the memories 128, and/or links to externally stored photos, videos, messages, websites, etc. from social media platforms, online video sharing platforms, etc. Furthermore, the user interface 124 may provide additional alerts, notifications, communications, etc., as discussed elsewhere herein. Moreover, the user interface 124 may be configured to provide guidance, look previews, and other information as projections upon the user device 120.

In some examples, the user interface 124 may generate and display an AR rendering of three-dimensional map of the user's face, and/or a selected makeup look as predicted to appear when applied to the user's face. For example, in some cases, the user interface 124 may overlay the AR rendering upon an image or video of the user's face as captured in real-time by sensors 122 of the user device 120 to illustrate a predicted appearance of a desired makeup look when applied to the user's face. In some examples, the user interface 124 may provide overlay upon an area of the user's face that is highlighted to illustrate that a cosmetic product should be applied to that area.

For example, when a cat-eye look is part of a user's desired makeup look (e.g., when a user selects a cat-eye look via the user interface 124 of the user device 120, the user interface 124 may generate an AR overlay including a trace of a cat-eye look upon the user's eyes as they appear in images captured by the user device 120 (e.g., by the sensors 122 of the user device 120). For instance, the user may apply an eyeliner product over the trace in order to apply the eyeliner product to the user's eyes to achieve the cat-eye look. This cat-eye look may be a part of the overlay projection. As another example, when a contour is part of the user's desired makeup look, the user interface 124 may generate an AR overlay highlighting certain areas of the user's cheekbones, chin, forehead, nose, etc. to which a contouring product should be applied in order to achieve the desired look. Accordingly, the user may apply a contouring product, such as a blush or bronzer, in the areas shown in the overlay projection to achieve the desired contoured look.

In some embodiments, the user interface 124 may generate and/or display an AR rendering of three-dimensional map of the user's face, and/or a selected makeup look as predicted to appear when applied to the user's face. For example, in some cases, the user interface 124 may overlay the AR rendering upon an image or video of the user's face as captured in real-time by sensors 122 of the user device 120 and/or sensors 142 of the smart mirror 140, to illustrate the appearance of the makeup look as applied to the user's face. In some examples, the user interface 124 may provide guidance to the user, e.g., such that an overlay upon an area of the user's face may be highlighted to illustrate that a cosmetic product should be applied to that area.

For example, the user interface 124 may overlay a trace of a cat-eye look upon the user's eyes as shown in an image or video of the user, such that the user may apply an eyeliner product over the trace in order to apply the eyeliner product to the user's eyes to achieve the cat-eye look. This cat-eye look may be a part of the overlay projection. As another example the user interface 124 may highlight certain areas of the user's cheekbones, chin, forehead, nose, etc. as shown in an image or video of the user. For instance, the user may apply a contouring product, such as a blush or bronzer, in the areas shown in the overlay projection to achieve a desired contoured look.

sensors 122 Memorie(s) 128 may include one or more forms of volatile and/or non-volatile, fixed and/or removable memory, such as read-only memory (ROM), electronic programmable read-only memory (EPROM), random access memory (RAM), erasable electronic programmable read-only memory (EEPROM), and/or other hard drives, flash memory, MicroSD cards, and others. Memorie(s) 128 may store an operating system (OS) (e.g., iOS, Microsoft Windows, Linux, UNIX, etc.) capable of facilitating the functionalities, apps, methods, or other software as discussed herein. The memories 128 may store instructions that, when executed by the processor(s) 127, cause the processor(s) 127 to receive input from a user as provided via the user interface 124 (e.g., via interactive user interface display screens), and send the received user input to the servers 110 and/or smart mirror 140 (e.g., via the network 130), in some cases responsive to a request for such user input from the servers 110 and/or smart mirror 140.

Moreover, in some examples, the memories 128 may store instructions that, when executed by the processor(s) 127, cause the processor(s) 127 to receive, from the servers 110 and/or smart mirror 140, indications of guidance to be provided to the user for applying one or more cosmetic products to achieve a look selected by the user, and may in turn provide the received guidance audible and/or visibly via the user interface 124 of the user device 120. Furthermore, in some examples, the memories 128 may store instructions that, when executed by the processor(s) 127, cause the processor(s) 127 to capture sensor data via one or more sensors 122, in some cases responsive to a request for particular sensor data from the servers 110 and/or smart mirror 140, and may send the captured sensor data to the servers 110 and/or smart mirror 140.

Moreover, in some examples, the memories 128 may store instructions that, when executed by the processor(s) 127, cause the processor(s) 127 to provide light to the face of the user via a light source 126, in some cases responsive to a request from the user device 120 to provide light to the face of the user. In some examples, the request may include a request for a particular lighting parameter, such as a particular level/intensity of light, or a particular warmth or color of light, and the processor(s) 127 may in turn cause the light source 126 to provide the requested level/intensity, color, warmth, etc. of light to the face of the user. Furthermore, in some examples, the instructions stored on the memorie(s) 128 may cause the processor(s) 127 to perform any or all of the steps of the method 900 and/or 1000 discussed above with respect to FIG. 9 and FIG. 10.

The smart mirror 140 may include a projector 141, one or more sensors 142, one or more user interfaces 143, one or more components 145, one or more light sources 146 configured to provide light to the face of the user, and/or an augmented reality module 147. Additionally, the smart mirror 140 may include a controller 148, including one or more processor(s) 150, as well as one or more computer memories 152. The projector 141 may be used to generate the overlay projection and to project an overlay onto a surface of the smart mirror 140. The smart mirror 140 may provide step-by-step audible and/or visual guidance to the user indicating, for instance, where to apply the cosmetic product via the overlay projection (for instance, including a visual indication of a location on the face of the user where the cosmetic product should be applied), how the cosmetic product should be blended with other products, patterns/shapes/motions to be used when applying the cosmetic product, etc. In some examples, the smart mirror 140 may provide this step-by-step guidance in real-time as the user applies or attempts to apply the cosmetic products to his or her face.

Generally speaking, the sensors 142 may be configured to capture real-time data associated with the face of a user before, during, and/or after a user applies a cosmetic product to the user's face. The sensors 142 may include, for instance, a camera (in some cases, a high resolution camera) and/or a depth sensor configured to capture data associated with the user's face, data associated with various cosmetic products to be applied to the user's face and/or their packaging, etc. Moreover, the sensors 142 may include sensors (e.g., the camera and/or the depth sensor, or additional or alternative sensors) configured to capture biometric data associated with the user, such as facial recognition data, fingerprint recognition data, iris recognition data, etc.

The user interface 143 may be configured to receive inputs and selections from the user of the smart mirror 140, and/or to provide audible or visual feedback to the user of the smart mirror 140, including instructions, guidance, tutorials, etc., associated with the user applying cosmetic products to the user's face for a particular makeup look selected by the user. For instance, the user interface 143 may be configured to receive an indication of a makeup look from a user and provide guidance associated with applying one or more cosmetic products to the facial features of the user in order to achieve a desired makeup look, based on real-time data captured by the sensors 142. In some embodiments, the user interface 143 may receive input from the user indicating a selection of a makeup look. The selected makeup look may be associated with various parameters and/or specifications that the smart mirror 140 may use to provide guidance to the user so that the user can apply cosmetic products to his or her face to achieve the selected look. Additionally, the user interface 143 may receive an image or a social media link from the user.

Moreover, in some examples, the user interface 143 may be configured to provide feedback associated with a selected makeup look after the selected makeup look is applied by the user. This feedback may include an indication of a level of precision based on the user's techniques and application skills. This precision level may range from a beginner level to an expert level. Based on the user's precision level, the user interface 143 may direct the user to resources to assist the user in improving their makeup application skills. These resources may include internal resources such as photos, videos, messages, etc. stored on the memories 152, and/or links to externally stored photos, videos, messages, websites, etc. from social media platforms, online video sharing platforms, etc. Furthermore, the user interface 143 may provide additional alerts, notifications, communications, etc., as discussed elsewhere herein. Moreover, the user interface 143 may be configured to provide guidance, look previews, and other information as projections upon the smart mirror 140.

The smart mirror 140 may also include an augmented reality (AR) module 147 configured to generate an overlay projection of cosmetic product application guidance that is projected (e.g., via the projector 141) onto a reflection of the face of the user as it appears in the smart mirror 140.

The AR module 147 may generate and display an AR rendering of three-dimensional map of the user's face, and/or a selected makeup look as predicted to appear when applied to the user's face. For example, in some cases, the AR module 147 may overlay the AR rendering upon an image or video of the user's face as captured in real-time by sensors 142 of the smart mirror 140 to illustrate a predicted appearance of a desired makeup look when applied to the user's face. In some examples, the AR module 147 may provide overlay upon an area of the user's face that is highlighted to illustrate that a cosmetic product should be applied to that area.

For example, when a cat-eye look is part of a user's desired makeup look (e.g., when a user selects a cat-eye look via the user interface 143 of the smart mirror 140 and/or the user interface 124 of the user device 120), the AR module 147 may generate an AR overlay including a trace of a cat-eye look upon the user's eyes as they appear reflected in the smart mirror 140. For instance, the user may apply an eyeliner product over the trace in order to apply the eyeliner product to the user's eyes to achieve the cat-eye look. This cat-eye look may be a part of the overlay projection. As another example, when a contour is part of the user's desired makeup look, the AR module 147 may generate an AR overlay highlighting certain areas of the user's cheekbones, chin, forehead, nose, etc. to which a contouring product should be applied in order to achieve the desired look. Accordingly, the user may apply a contouring product, such as a blush or bronzer, in the areas shown in the overlay projection to achieve the desired contoured look.

The controller 148 may include the one or more processors 150 and the one or more memories 152. The memories may include one or more forms of volatile and/or non-volatile, fixed and/or removable memory, such as read-only memory (ROM), electronic programmable read-only memory (EPROM), random access memory (RAM), erasable electronic programmable read-only memory (EEPROM), and/or other hard drives, flash memory, MicroSD cards, and others. Memorie(s) 152 may store an operating system (OS) (e.g., iOS, Microsoft Windows, Linux, UNIX, etc.) capable of facilitating the functionalities, apps, methods, or other software as discussed herein.

Generally speaking, the memories 152 may store instructions that, when executed by the processor(s) 150, cause the processors 150 to receive an indication of a makeup look from a user (e.g., from a user interface 143 of the smart mirror 140, or from a user interface 124 of an associated user device 120), and instructions that, when executed by the controller 148, cause the smart mirror 140 to generate and provide guidance via the projector 141, user interface 143, and/or AR module 147, for a user to apply a cosmetic product based on the selected makeup look.

Furthermore, the memories 152 may store instructions that, when executed by the processor(s) 150, cause the processor(s) 150 to analyze images associated with the user's cosmetic product application to identify the user's techniques in applying the cosmetic products to the user's face. For instance, the memories 152 may store instructions that, when executed by the processor(s) 150, cause the processor(s) 150 to capture image data (e.g., via the sensors 142 and/or sensors 122) associated with the user's makeup application, and analyze the image data associated with the user's application techniques, e.g., to identify any techniques used by the user in applying a cosmetic look and/or a level of precision associated with any of the techniques used by the user in applying the cosmetic look. For instance, the memories 152 may store instructions that, when executed by the processor(s) 150, cause the processor(s) 150 to analyze images captured by the sensors 142, or send the images captured by the sensors to a server 110, to be analyzed by a machine learning model 116, in order to identify or predict a level of precision associated with the images. In some examples, the identified or predicted level of precision may be determined based on the techniques used by the user in applying the cosmetic look (e.g., the types of motions made by the user's hands or applicators, the speed or intensity with which the user's hands or applicators, etc.), and/or may be determined based on the user's finished makeup look (e.g., whether or to what extent the finished makeup look is determined to be aesthetically pleasing in general, whether or to what extent the finished makeup look flatters the user's features and/or complexion, whether or to what extent the finished makeup look matches a desired makeup look selected by the user, whether or to what extent the user followed guidance provided by the smart mirror 140 and/or the user device 120, etc.).

Furthermore, the instructions stored on the memories 152 may cause the processors 150 to analyze real-time sensor data captured by the sensors 142 (and/or external sensors, such as sensors 122 of a user device 120) in order to generate an overlay projection or in some examples, a three-dimensional map associated with the user's face and identify the locations of one or more facial features (e.g., eyes, eyelids, eyebrows, eyelashes, cheeks, cheekbones, nose, lips, chin, etc.) of the user's face on the overlay projection or three-dimensional map.

Additionally, the instructions stored on the memories 152 may cause a smart mirror 140 to provide audible or visible feedback, guidance, or tutorials to the user in real-time as the user applies the makeup look, or may send such feedback, guidance, or tutorials to another device (such as the user device 120) for display via the user interface of that device (e.g., the user interface 143 of the smart mirror 140 or the user interface 124 of the user device 120). The processor 150 of the smart mirror 140 may also analyze images or social media links to determine parameters and/or specifications that the smart mirror 140 may use and provide guidance to the user for applying cosmetic products to his or her face to achieve a selected look. For instance, these specifications may include types of makeup applied to each area of the face, heaviness of makeup applied to each area of the face, particular patterns, shapes, or borders of makeup applied to each area of the face, layers of makeup applied to each area of the face, etc.

In some examples, the instructions stored on the memories 152 may cause the processors 150 to determine that the user has sufficiently followed the overlay projection and may accordingly automatically proceed to a subsequent step in the guidance. Additionally, in some examples, the processors 150 of the smart mirror 140 may determine that a user is having difficulty completing a step (e.g., based on the way that the cosmetic products are applied to the user's face, and/or based on the way that the user is attempting to apply the cosmetic products to the user's face), and may accordingly provide additional guidance via the user interface 143 (e.g., additional guidance for removing an incorrectly applied cosmetic product, additional guidance regarding recommended techniques, additional guidance regarding adjustments to actions being performed by the user, additional guidance for selecting a less challenging look, etc.).

Moreover, the instructions stored on memory 152 may cause the controller 148 to adjust the feedback based on conditions associated with the user's skin as detected in real-time, e.g., based on data captured by the sensors 142 of the smart mirror 140. For instance, the instructions stored on the memories 152 may cause the processor(s) 150 to analyze image data captured by the sensors 142 to detect application techniques of the user and may, for instance, cause the controller 148 to adjust the feedback and/or overlay projection to guide the user's actions such that additional cosmetic products or additional coats of cosmetic products are applied to specific areas of the user's face. Furthermore, in some examples, the instructions stored on the memories 152 may cause the processor(s) 150 to analyze image data captured by the sensors 142 to detect application progress and may, in some cases, cause the controller 148 to adjust the feedback and/or overlay projection to guide the user to cease applying the cosmetic products or apply the cosmetic products in a manner to avoid further misapplications of the cosmetic product.

Furthermore, in some examples, the instructions stored on the memories 152 may cause the processor(s) 150 and/or the controller 148 to perform any or all of the steps of the method 900 and/or 1000 discussed below with respect to FIG. 9 and FIG. 10.

FIG. 2 depicts a user's initial application of a cosmetic product 202 to a user's face 204, according to some embodiments.

FIG. 3 depicts an exemplary handheld device 302 (e.g., one of the devices 120 discussed above with respect to FIG. 1) displaying an indication of a user's level of precision 308 in applying a cosmetic product to the user's face 304 prior to receiving cosmetic product application guidance, according to some embodiments. As shown in FIG. 3, the level of precision 308 may be shown at the bottom of the user's screen or display, as shown on the handheld device 302. The level of precision 308 may also be displayed on any other portion of the screen or display of the user's handheld device 302 or any other device. The level of precision 308 may range from a beginner level, as depicted in FIG. 3, to an expert level.

FIG. 4 depicts an exemplary handheld device 402 (e.g., one of the devices 120 discussed above with respect to FIG. 1) displaying an overlay projection 403 of cosmetic product application guidance over an image of a user's face 404, according to some embodiments. For instance, the handheld device 402 may display the overlay projection over live images of the user's face 404 captured by the sensors of the handheld device 402 in real-time during or before the user 406 applies the makeup or cosmetic product to the user's face 404. In some examples, the handheld device 402 may provide user prompts 411 such as, “Use this overlay projection as a guide,” as depicted in FIG. 4. For instance, these prompts 411 may be provided to the user 406 visually via a display of the handheld device 402 and/or may be provided audibly via a speaker of the handheld device 402.

FIG. 5 depicts an exemplary mirror 509 upon which an overlay projection 503 of cosmetic product application guidance is displayed over a reflection of a user's face 504, according to some embodiments. FIG. 5 depicts an example of the overlay projection 503 being superimposed on the user's reflection, such as via the user interface 143 of the smart mirror 140 of FIG. 1. As depicted in FIG. 5, the mirror 509 reflects the user 506 viewing the user's face 504 prior to applying or reapplying cosmetic products. The mirror 509 may provide guidance that appears directly on the mirror 509, including some guidance that is superimposed upon the user's face 504 as it appears reflected in the mirror 509 via a user interface, such as the user interface 143 of the smart mirror 140 of FIG. 1.

For instance, as shown in FIG. 5, the overlay projection 503, via a projector such as the projector 141 of the smart mirror 140 of FIG. 1, includes visible guidance for applying a makeup look, which includes guidelines indicating where exactly to apply a particular cosmetic product on the user's face 504. This projector may superimpose the overlay projection 503 upon the user's face 504 as it appears in the mirror 509, such that a user 506 may trace the guidelines in the area indicated by overlay projection 503 with a cosmetic applicator to apply the cosmetic product to the user's face 504. As shown in FIG. 5, the overlay projection 503, via the projector, may include eye, nose, and lip tracing guidance. Guidance which may be superimposed on the reflection of the user 506 in the mirror 509. The overlay projection 503, via the projector, may assist in improving the user's accuracy and functionality when applying cosmetic products and makeup, thus improving the user's precision level over time.

FIG. 6 depicts an exemplary user applying a cosmetic product 602 to the user's face 604 based on cosmetic product application guidance provided via an overlay projection 603, according to some embodiments. As shown in FIG. 6, the overlay projection 603 may provide the user with guidelines which the user may be able to trace over or follow for an improved makeup look.

FIG. 7 depicts a user's final look after applying a cosmetic product to the user's face 704 based on cosmetic product application guidance a user's final look after application of cosmetic product to the user's face 704 and after following the overlay projection 603 of cosmetic product application guidance, according to some embodiments.

FIG. 8 depicts an exemplary handheld device 802 displaying an indication of a user's new level of precision 808 in applying a cosmetic product after receiving cosmetic product application guidance, according to some embodiments. The new precision level 808 may be depicted at the bottom of the user's screen or display on any other portion of the screen or display of the user's handheld device 802 or any other device. The precision level 808 may range from a beginner level to an expert level.

Example Methods

FIG. 9 depicts an example flow diagram of an exemplary method 900 for generating and providing cosmetic product application guidance in real-time as a user applies cosmetic products, according to some embodiments. One or more steps of the method 900 may be implemented as instructions stored on a computer-readable memory (e.g., memory 114, memory 128, memory 152, etc.) and executable on one or more processors (e.g., processor(s) 112, processor(s) 127, processor(s) 150, etc.).

The method 900 may include, at block 901, capturing the one or more images of the face of the user. The user's face may be captured via the system's one or more sensors such as sensors 122 of the user device 120 and sensors 142 of the smart mirror 140 discussed above with respect to FIG. 1.

At block 902, the method 900 may include analyzing the one or more images of the face of the user as the user applies one or more cosmetic products to at least one facial feature of the user to identify techniques used by the user.

At block 903, the method 900 may include evaluating the techniques used by the user using a precision detection machine learning model, in order to determine a level of precision associated with each technique used by the user. The determined level of precision may also include a range from a beginner level to an expert level. The precision detection machine learning model, such as the machine learning model 116 discussed above with respect to FIG. 1, may be trained using training data of previous cosmetic applications, e.g., data associated with previous instances in which the user applied cosmetic products, and/or data associated with previous instances in which other users applied cosmetic products. The precision detection machine learning model may determine a level of precision associated with the techniques used by the user. In some embodiments, the training data of previous cosmetic applications may include one or more historical images and/or videos of users performing various techniques, labeled with techniques shown in each image or video, and/or labeled with levels of precision associated with each identified technique.

In some embodiments, the level of precision determined by the precision detection machine learning model may be based on at least one of a set of criteria including symmetry and balance, blending and color matching, precision of lines and edges, and overall aesthetic appeal. The precision detection machine learning model may analyze sensor data to determine an evenness of an application as the user applies the one or more cosmetic products. The level of precision determined by the precision detection machine learning model may also be based on blending and color matching. The precision detection machine learning model may evaluate a smoothness of a transition between at least one color and at least one texture as the user applies the one or more cosmetic products. In some embodiments the level of precision determined by the precision detection machine learning model may be based on precision of lines and edges. The precision detection machine learning model may analyze sensor data to determine an accuracy and a sharpness of lines and edges as the user applies the one or more cosmetic products. In some examples, the level of precision determined by the precision detection machine learning model may be based on overall aesthetic appeal. The precision detection machine learning model may also evaluate a harmony and an overall visual impact of a completed cosmetic application. In each of these examples, the training data may include indications of measurements and evaluations associated with historically identified techniques.

The method 900 may also include, at block 904, generating an overlay projection of cosmetic product application guidance associated with the one or more images of the face of the user in real-time, based on the user's determined level of precision. Additionally, the method 900 may include displaying, at block 905 the overlay projection of cosmetic product application guidance upon the user's face in real-time, e.g., via the user interface 124 of the user device 120 or the user interface 143 of the smart mirror 140 discussed above with respect to FIG. 1. For instance, the overlay projection may be superimposed upon one or more images and/or videos of the face of the user, or a reflection of the face of the user as it appears in a mirror, in real-time. The overlay projection may be used as a guide to assist in improving the user's skills in applying makeup and other cosmetic products. In some examples, the overlay projection may provide step-by-step instructions and visual cues appropriate to the user's determined level of precision, in order to help the user apply the makeup accurately and efficiently and improve his or her level of precision. Using the guidelines from the overlay projection as shown superimposed upon the user's face in the images, videos, or mirror, the user may trace over or follow the lining of the overlay projection for an improved makeup look. The overlay projection may be designed to guide the user's makeup application by providing feedback.

In some examples, in addition to providing the overlay projection of the cosmetic product application guidance, the method 900 may include displaying or providing links to relevant tutorials or other online resources appropriate for the determined level of precision.

In some examples, the method 900 may also include, after displaying the overlay projection of cosmetic product application guidance, capturing one or more new images of the face of the user, during or after the user applies the cosmetic products based on the cosmetic product application guidance, and analyzing the new images using the precision detecting machine learning model discussed with respect to block 903, e.g., to identify new techniques used by the user or changes to previous techniques used by the user, and/or to identify or predict a new or updated level of precision associated with the user's new techniques. For instance, the method 900 may include displaying an indication of the new or updated level of precision via a user interface. Moreover, he method 900 may include analyzing the one or more new images of the face of the user, during or after the user applies the cosmetic products based on the cosmetic product application guidance, to determine whether the user is following the guidance from the overlay projection, and/or to what extent the user is following the guidance from the overlay projection.

The method 900 may also include generating a new overlay projection of cosmetic product application guidance in real-time, based on the new level of precision determined by the precision detection machine learning model, and/or based on determining whether and/or to what extent the user followed the previous guidance. For instance, the new guidance may include additional guidance for any techniques with which the user is struggling, and/or may remove guidance associated with techniques in which the user is now proficient.

FIG. 10 depicts example flow diagram of an exemplary method 1000 for generating and providing cosmetic product application guidance for a user based on a user's desired makeup look, according to some embodiments. One or more steps of the method 1000 may be implemented as instructions stored on a computer-readable memory (e.g., memory 114, memory 128, memory 152, etc.) and executable on one or more processors (e.g., processor(s) 112, processor(s) 127, processor(s) 150, etc.).

The method 1000 may include, at block 1001, receiving an image of a face depicting a user's desired makeup look (for instance, as selected by a user, uploaded by a user, provided as a URL link by the user, etc., via a user interface 124 and/or user interface 143 discussed above with respect to FIG. 1).

At block 1002, the method 1000 may include capturing, using the one or more sensors, such as sensors 122 of the user device and/or sensors 142 of the smart mirror 140 discussed with respect to FIG. 1, one or more images of the face of the user.

At block 1003, the method 1000 may also include comparing the image of the face depicting the user's desired makeup look to the one or more images or videos of the face of the user as the user attempts to replicate the desired makeup look provided at block 1001, in order to identify techniques used by the user in attempting to replicate the desired makeup look. In some examples, the user's makeup application process is analyzed to determine if the user's desired makeup look and current look are similar. The user's application of eye makeup such as eyeshadow, eyeliner, eye lashes, etc., or lipstick, for example, may be compared to the user's desired makeup look to determine accuracy in application of the eye makeup and lipstick.

Additionally, at block 1004, the method 1000 may include evaluating the techniques used by the user using a precision detection machine learning model, wherein the precision detection machine learning model determines a level of precision associated with the techniques used by the user. The determined level of precision may also include a range from a beginner level to an expert level, and, in some examples, may be based on whether the user successfully replicates the desired makeup look, and/or the extent to which the user replicates the desired makeup look. The precision detection machine learning model, such as the machine learning model 116 discussed above with respect to FIG. 1, may be trained using training data of previous cosmetic applications, e.g., data associated with previous instances in which the user applied cosmetic products, and/or data associated with previous instances in which other users applied cosmetic products. The precision detection machine learning model may determine a level of precision associated with the techniques used by the user. In some embodiments, the training data of previous cosmetic applications may include one or more historical images and/or videos of users performing various techniques, labeled with techniques shown in each image or video, and/or labeled with levels of precision associated with each identified technique.

In some embodiments, the level of precision determined by the precision detection machine learning model may be based on at least one of a set of criteria which may include symmetry and balance, blending and color matching, smoothness of transition, precision of line and edges, sharpness, and more. The precision detection machine learning model may analyze sensor data to determine an evenness of an application as the user applies the makeup. The level of precision determined by the precision detection machine learning model may also be based on blending and color matching. The precision detection machine learning model may evaluate a smoothness of a transition between colors and textures as the user applies the makeup. In some embodiments the level of precision determined by the precision detection machine learning model may be based on precision of lines and edges. The precision detection machine learning model may analyze sensor data to determine an accuracy and a sharpness of lines and edges as the user applies the one or more cosmetic products. In some examples, the level of precision determined by the precision detection machine learning model may be based on overall aesthetic appeal, wherein harmony and an overall visual impact of a completed cosmetic application are analyzed. In each of these examples, the training data may include indications of measurements and evaluations associated with historically identified techniques.

The method 1000 may also include, at block 1004, generating an overlay projection of cosmetic product application guidance associated with the one or more images of the face of the user in real-time, based on the user's determined level of precision. Additionally, the method 1000 may include displaying, at block 1006 the overlay projection of cosmetic product application guidance upon the user's face in real-time, e.g., via the user interface 124 of the user device 120 or the user interface 143 of the smart mirror 140 discussed above with respect to FIG. 1. For instance, the overlay projection may be superimposed upon one or more images and/or videos of the face of the user, or a reflection of the face of the user as it appears in a mirror, in real-time. The overlay projection may be used as a guide to assist in improving the user's skills in applying makeup and other cosmetic products in accordance with the user's determined level of precision in order to achieve a desired makeup look. In some examples, the overlay projection may provide step-by-step instructions and visual cues to help the user apply the makeup accurately and efficiently. Using the guidelines from the overlay projection as shown superimposed upon the user's face in the images, videos, or mirror, the user may trace over or follow the lining of the overlay projection for an improved makeup look. The overlay projection may be designed to guide the user's makeup application by providing feedback.

In some examples, in addition to providing the overlay projection of the cosmetic product application guidance, the method 1000 may include displaying or providing links to relevant tutorials or other online resources appropriate for the determined level of precision.

In some examples, the method 1000 may also include, after displaying the overlay projection of cosmetic product application guidance, capturing one or more new images of the face of the user, during or after the user applies the cosmetic products based on the cosmetic product application guidance, and analyzing the new images using the precision detection machine learning model discussed with respect to block 1004, e.g., to identify new techniques used by the user or changes to previous techniques used by the user, and/or to identify or predict a new or updated level of precision associated with the user's new techniques. For instance, the method 900 may include displaying an indication of the new or updated level of precision via a user interface.

Moreover, in some examples, identifying and/or predicting the new or updated level of precision may include analyzing the one or more new images of the face of the user, during or after the user applies the cosmetic products based on the cosmetic product application guidance, to determine whether the user is following the guidance from the overlay projection, and/or to what extent the user is following the guidance from the overlay projection. Furthermore, identifying and/or predicting the new or updated level of precision may include analyzing the one or more new images of the face of the user, during or after the user applies the cosmetic products based on the cosmetic product application guidance, to determine whether the use has accurately replicated the desired makeup look, and/or an extent to which the user has accurately replicated the desired makeup look.

The method 1000 may also include generating a new overlay projection of cosmetic product application guidance in real-time, based on the new level of precision determined by the precision detection machine learning model, and/or based on determining whether and/or to what extent the user followed the previous guidance. For instance, the new guidance may include additional guidance for any techniques with which the user is struggling, and/or may remove or provide less guidance associated with techniques in which the user is now proficient.

Additional Considerations

The following additional considerations apply to the foregoing discussion. Throughout this specification, plural instances may implement operations or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of the individual operations may be performed concurrently, and nothing requires that the operations be performed in the order illustrated. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.

Unless specifically stated otherwise, discussions herein using words such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” or the like may refer to actions, elements, or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

As used herein any reference to “one embodiment” or “an embodiment” or “some embodiments” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in one embodiment” or “in some embodiments” in various places in the specification are not necessarily all referring to the same embodiment.

As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a non-exclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

In addition, use of “a” or “an” is employed to describe elements and components of the embodiments herein. This is done merely for convenience and to give a general sense of the invention. This description should be read to include one or at least one and the singular also includes the plural unless it is obvious that it is meant otherwise.

Upon reading this disclosure, those of skill in the art will appreciate still additional alternative structural and functional designs for systems that provide users with an overlay projection guide to assist in improving the user's skills in applying makeup and other cosmetic products, and/or systems, methods, and/or techniques associated therewith. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments are not limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which will be apparent to those skilled in the art, may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope defined in the appended claims.

Aspects

• • 1. A system comprising: one or more sensors configured to capture, in real-time, one or more images of a face of a user; one or more processors; and one or more memories storing non-transitory computer-readable instructions that, when executed by the one or more processors, cause the one or more processors to: capture, using the one or more sensors, the one or more images of the face of the user; analyze the one or more images of the face of the user as the user applies one or more cosmetic products to at least one facial feature of the user to identify techniques used by the user; evaluate the techniques used by the user using a precision detection machine learning model wherein the precision detection machine learning model is trained using training data of previous cosmetic applications, and wherein the precision detection machine learning model determines a level of precision associated with the techniques used by the user; generate in real-time, based on the determined level of precision determined by the precision detection machine learning model, an overlay projection of cosmetic product application guidance associated with the one or more images of the face of the user; and using an augmented reality module, display the overlay projection of cosmetic product application guidance onto one of: the one or more images of the face of the user, a reflection of the face of the user as it appears in a mirror in real-time, or the face of the user in real-time.
  • 2. The system of aspect 1, wherein the non-transitory computer-readable instructions, when executed by the one or more processors, further cause the one or more processors to provide relevant content to the user based on the determined level of precision.
  • 3. The system of any one of aspects 1 or 2, wherein the one or more sensors include one or more sensors of a camera or smart device, wherein the camera or smart device captures the one or more images of the face of the user using high-resolution.
  • 4. The system of any one of aspects 1-3, wherein the training data of previous cosmetic applications includes one or more images of users performing historically identified techniques and historical levels of precision associated with each historically identified technique.
  • 5. The system of any one of aspects 1-4, wherein the determined level of precision includes a range from a beginner level to an expert level.
  • 6. The system of any one of aspects 1-5, wherein the level of precision determined by the precision detection machine learning model is based on at least one of a set of criteria including: symmetry and balance, wherein the precision detection machine learning model analyzes sensor data to determine an evenness of an application as the user applies the one or more cosmetic products, blending and color matching, wherein the precision detection machine learning model evaluates a smoothness of a transition between at least one color and at least one texture as the user applies the one or more cosmetic products, precision of lines and edges, wherein the precision detection machine learning model analyzes sensor data to determine an accuracy and a sharpness of lines and edges as the user applies the one or more cosmetic products, or overall aesthetic appeal, wherein the precision detection machine learning model evaluates a harmony and an overall visual impact of a completed cosmetic application, wherein the training data includes indications of measurements and evaluations associated with historically identified techniques.
  • 7. The system of any one of aspects 1-6, wherein the non-transitory computer-readable instructions, when executed by the one or more processors, further cause the one or more processors to: after displaying the overlay projection of cosmetic product application guidance, capture, using the one or more sensors, one or more new images of the face of the user; analyze the one or more new images of the face of the user as the user applies the one or more cosmetic products to the at least one facial feature of the user to identify new techniques used by the user; evaluate the new techniques used by the user using the precision detection machine learning model, wherein the precision detection machine learning model determines a new level of precision associated with the new techniques used by the user; generate in real-time, based on the new level of precision determined by the precision detection machine learning model, a new overlay projection of cosmetic product application guidance associated with the one or more new images of the face of the user; and using the augmented reality module, display the new overlay projection of cosmetic product application guidance onto one of: the one or more new images of the face of the user, the reflection of the face of the user as it appears in the mirror in real-time, or the face of the user in real-time.
  • 8. The system of any one of aspects 1-7, wherein the one or more sensors, one or more processors, and one or more memories are part of a smart device and wherein the smart device includes a display screen.
  • 9. A system comprising: one or more sensors configured to capture, in real-time, one or more images of a face of a user; one or more processors; and one or more memories storing non-transitory computer-readable instructions that, when executed by the one or more processors, cause the one or more processors to: receive an image of a face depicting a user's desired makeup look; capture, using the one or more sensors, the one or more images of the face of the user; compare the image of the face depicting the user's desired makeup look to the one or more images of the face of the user, in order to identify techniques used by the user; evaluate the techniques used by the user using a precision detection machine learning model wherein the precision detection machine learning model is trained using training data of previous cosmetic applications, and wherein the precision detection machine learning model determines a level of precision associated with the techniques used by the user; generate in real-time, based on the determined level of precision determined by the precision detection machine learning model, an overlay projection of cosmetic product application guidance for achieving the user's desired makeup look; and using an augmented reality module, display the overlay projection of cosmetic product application guidance onto one of: the one or more images of the face of the user, a reflection of the face of the user as it appears in a mirror in real-time, or the face of the user in real-time.
  • 10. The system of aspect 9, wherein the non-transitory computer-readable instructions, when executed by the one or more processors, further cause the one or more processors to provide relevant content to the user based on the determined level of precision.
  • 11. The system of any one of aspects 9-10, wherein the one or more sensors include one or more sensors of a camera or smart device, wherein the camera or smart device captures the one or more images of the face of the user using high-resolution.
  • 12. The system of any one of aspects 9-11, wherein the training data of previous cosmetic applications includes one or more images of users performing historically identified techniques and historical levels of precision associated with each historically identified technique.
  • 13. The system of any one of aspects 9-12, wherein the determined level of precision includes a range from a beginner level to an expert level.
  • 14. The system of any one of aspects 9-13, wherein the level of precision determined by the precision detection machine learning model is based on at least one of a set of criteria including: symmetry and balance, wherein the precision detection machine learning model analyzes sensor data to determine an evenness of an application as the user applies the one or more cosmetic products, blending and color matching, wherein the precision detection machine learning model evaluates a smoothness of a transition between at least one color and at least one texture as the user applies the one or more cosmetic products, precision of lines and edges, wherein the precision detection machine learning model analyzes sensor data to determine an accuracy and a sharpness of lines and edges as the user applies the one or more cosmetic products, or overall aesthetic appeal, wherein the precision detection machine learning model evaluates a harmony and an overall visual impact of a completed cosmetic application, wherein the training data includes indications of measurements and evaluations associated with historically identified techniques.
  • 15. The system of any one of aspects 9-14, wherein the one or more sensors, one or more processors, and one or more memories are part of a smart device and wherein the smart device includes a display screen.
  • 16. A method comprising: capturing, using one or more sensors, one or more images of a face of a user; analyzing, by one or more processors, the one or more images of the face of the user as the user applies one or more cosmetic products to at least one facial feature of the user to identify techniques used by the user; evaluating, by the one or more processors, the techniques used by the user using a precision detection machine learning model wherein the precision detection machine learning model is trained using training data of previous cosmetic applications, and wherein the precision detection machine learning model determines a level of precision associated with the techniques used by the user; generating, by the one or more processors, in real-time, based on the determined level of precision determined by the precision detection machine learning model, an overlay projection of cosmetic product application guidance associated with the one or more images of the face of the user; and using an augmented reality module, displaying, by the one or more processors, the overlay projection of cosmetic product application guidance onto one of: the one or more images of the face of the user, a reflection of the face of the user as it appears in a mirror in real-time, or the face of the user in real-time.
  • 17. The method of aspect 16, wherein the method further includes: after displaying, by the one or more processors, the overlay projection of cosmetic product application guidance, capture, using the one or more sensors, one or more new images of the face of the user; analyze, by the one or more processors, the one or more new images of the face of the user as the user applies the one or more cosmetic products to the at least one facial feature of the user to identify new techniques used by the user; evaluating, by the one or more processors, the new techniques used by the user using the precision detection machine learning model, wherein the precision detection machine learning model determines a new level of precision associated with the new techniques used by the user; generate, by the one or more processors, in real-time, based on the new level of precision determined by the precision detection machine learning model, a new overlay projection of cosmetic product application guidance associated with the one or more new images of the face of the user; and using the augmented reality module, displaying, by the one or more processors, the new overlay projection of cosmetic product application guidance onto one of: the one or more new images of the face of the user, the reflection of the face of the user as it appears in the mirror in real-time, or the face of the user in real-time.
  • 18. The method of any one of aspects 16-17, wherein the method further includes, provide relevant content to the user based on the determined level of precision.
  • 19. The method of any one of aspects 16-18, wherein capturing one or more images of the face of the user includes utilizing one or more sensors of a camera or smart device wherein the camera or smart device captures the one or more images of the face of the user using high-resolution.
  • 20. The method of any one of aspects 16-19, wherein the training data of previous cosmetic applications includes one or more images of users performing historically identified techniques and historical levels of precision associated with each historically identified technique.