Methods and Systems to Help Detect Identity Fraud

Description

RELATED APPLICATION DATA

This application is a division of copending application Ser. No. 12/114,612, filed May 2, 2008 (published as US20080208849), which is a division of abandoned application Ser. No. 11/613,891, filed Dec. 20, 2006 (published as US20070162761), which claims priority benefit to provisional application 60/753,652, filed Dec. 23, 2005.

Some of the subject matter herein is related to that in other patent applications, including Ser. No. 10/723,240, filed Nov. 26, 2003 (published as US20040213437); Ser. No. 10/979,770, filed Nov. 1, 2004 (now U.S. Pat. No. 7,314,162); and Ser. No. 11/132,724, filed May 18, 2005 (published as US20050288952).

TECHNICAL FIELD

The technology detailed herein generally relates to methods and systems to aid in verifying a person's identity, e.g., in connection with applying for an identity document (such as a passport or driver's license), or in connection with qualifying to enter a secured area (such as at an airport).

BACKGROUND

Traditionally, applicants for identity documents have been required to present only a few items of collateral identification, such as a birth certificate, a social security card, and/or a study body ID card. (Such collateral documents are sometimes termed “breeder documents;” a fuller list of commonly-accepted breeder documents is detailed in application Ser. No. 10/979,770, published as US20070205266.) With the proliferation of low-cost and high-quality scanning and printing technologies, as well as simple image editing software, such breeder documents have become easier to counterfeit. Thus, there is a need for techniques by which the identity of an applicant can more reliably be determined.

Application Ser. No. 10/979,770 notes that the risk of identity fraud in the issuance of ID documents varies, with some types of breeder documents being more reliable in establishing a person's identity (e.g., US passports) than other types of breeder documents (e.g., student body ID cards). Data on the incidence of discovered fraud can be collected, and correlated back to the types of breeder documents submitted in each case, e.g., using factor analysis. This historical data permits a risk score to be generated for each new applicant, based on the particular types of breeder documents he or she presents. Applicants with relatively high breeder document risk scores can be scrutinized relatively more closely than applicants with relatively low risk scores. Such techniques allow security personnel to focus their efforts where they will do the most good.

Application Ser. No. 11/132,724 (published as US20050288952) notes that some parts of the applicant enrollment process can be performed from the applicant's home. A state Department of Motor Vehicles (DMV), for example, may have a web site through which an applicant for a driver's license can enter their name, address, birth date, hair color, organ donor preferences, and other background information. Scans of breeder documents that the applicant intends to present (e.g., birth certificate and passport) can also be submitted from home. In some systems the applicant may even be allowed to submit a proposed portrait photograph for printing on their license. This data-entry web session can conclude by allowing the applicant to schedule an appointment to visit a nearby DMV office to complete the enrollment and license issuance process.

By receiving this applicant information in advance, the DMV can undertake more thorough vetting of an applicant's identity than if they simply appear at the DMV office. Such vetting generally involves researching the applicant and his/her purported identity, and checking any breeder document data, to make sure that nothing appears amiss. For example, the DMV may check third party databases, such as credit bureaus, telephone directories, social security databases, etc., to verify that the information submitted by the applicant, and the information represented by the breeder documents, is consistent with data maintained by these third parties. Any portrait photograph submitted by the applicant can likewise be checked against an archive of previous driver license images to determine whether a person of similar appearance has already been issued a driver license. If these checks give any ground for suspicion, the DMV can contact the applicant to solicit further information. If issues are not satisfactorily addressed prior to the appointment, the appointment may be canceled.

Published application US20040213437 details technologies by which the photograph of a new applicant for a driver license can be checked against a database of photographs on previously-issued driver licenses. If a suspected match is found, the circumstances can be investigated to determine whether the applicant may be engaged in fraud.

Application Ser. No. 10/979,770 details how a risk score may be generated, to give an indication of the relative possibility of fraud associated with a given applicant (e.g., by considering past fraud experiences correlated with different types of breeder documents). Other risk scoring techniques are known the art. Examples are shown in published applications and patents such as US20030052768 (a trusted-traveler card system, in which trust is scored based on various factors, including how long the traveler has lived at a given address); US20030099379 and US20030115459 (various ID card attributes are sensed, and facial features and third party databases are checked, to yield a score by which confidence in a person's asserted identity is assessed); US20030216988 (the validity of an applicant's telephone number is checked, and combined with other indicators, to produce a risk score for a proposed transaction); US20040059953 (a transportation worker identity card system in which various personal data is collected from the applicant and checked against third party databases, to determine if confidence in the asserted identity exceeds a threshold); US20040064415 (a traveler's identity is checked by reference to various public and private databases—which may include birth certificate data, social security number data, and INS records—and a resulting score is produced); US20040153663 (in processing a change-of-address request, a credit card company compares demographics of the applicant's previous and new neighborhoods, e.g., average income, average net worth, percentage of renters, etc., looking for unexpected disparities; a polynomial equation is applied to compute an associated risk score); US20040230527 (before completing a wired money transfer, circumstances of the transfer are compared against historical data and referred to third party evaluation services, to generate a score indicative of the risk of charge-back); US20040245330 (before completing a financial transaction, various parameters are considered and contribute—positively or negatively—to a net score, which is used to determine whether the transaction should proceed); US20050039057 (during enrollment, a person is questioned re opinions and trivial facts, e.g., “I carry my car keys in my (a) pocket; (b) purse; (c) briefcase; (d) backpack”, “The phone number of a childhood friend is XXX-YYY-ZZZZ,” and the given answers are stored in a database; when the person's identity is later to be checked, a random subset of these questions is posed, some with different weightings, until a given confidence of identity is met); US20050132235 and US20050171851 (a user's speech is biometrically analyzed to determine degree of match against earlier-captured voice data, to yield an identification factor; this can be combined with other checks to generate a score indicating confidence in the speaker's identity); and U.S. Pat. No. 5,679,938 and U.S. Pat. No. 5,679,940 (different attributes of a financial check and an associated transaction are weighted to compute a score indicating degree of confidence that the check will be honored).

U.S. Pat. No. 6,513,018 details empirically-based scoring technologies employed by Fair, Isaac and Company in computing credit scores. Particularly detailed are arrangements by which different applicant characteristics are either positively- or negatively-correlated with certain performance results, and methods for determining and applying such correlative information. U.S. Pat. No. 6,597,775 details Fair, Isaac's extrapolation and improvement of such methodologies to predictive modeling, and mitigation, of telecommunications fraud.

Many ID verification systems rely on challenge-response testing concerning information that could become known to an imposter, e.g., by stealing a person's wallet or purse, or by removing mail from a mailbox. This gives rise to systems based on “out-of-wallet” information—the most common of which is “mother's maiden name.” More elaborate “out-of-wallet” systems for confirming identity are detailed, e.g., in some of the patents and publications referenced above, as well as in the patent publications such as: US20040189441 (which checks identity by testing applicant's knowledge of inherent attributes—such as mother's maiden name, as well as voluntary attributes—such as favorite color; the techniques provide some error tolerance in assessing answers, e.g., answer of “Smith” vs. expected answer of “Mr. Smith); and 20040205030 (provides a lengthy catalog of “out-of-wallet” information that may be used in confirming applicant identity, e.g., name of a signatory on a particular document, alimony payments, surgical records, medication currently prescribed, judicial records, etc.).

Some identity-verification systems employ multi-factor approaches. An example is US20050154924, which validates a user based on a collection of user-provided factors, such as a combination of ‘what the user knows’ (e.g., knowledge-based data), ‘who the user is’ (e.g., biometric-based data), ‘what the user possesses’ (e.g., token-based data), ‘where the user is’ (i.e., location-based data), and when the user is seeking validation (i.e., time-based data).

Biometrics can be useful in checking identity. But biometrics can only check for a match between an earlier-collected and presently-collected set of data. Unless there is confidence about the identity of the person from whom the earlier biometric data was collected, such technologies are of limited utility.

BRIEF DESCRIPTIONS OF THE DRAWINGS

FIGS. 1-11 illustrate information that can be used in testing an applicant to help determine relative confidence in the applicant's asserted identity.

DETAILED DESCRIPTION

For expository convenience, most of the following detailed description focuses on one particular application of applicants' technology: verifying the identity of an applicant for a driver license. It will be recognized that such technologies can likewise be employed to help verify the identity of persons in myriad other contexts.

The reader is presumed to be familiar with driver license issuance systems and procedures. (The commonly-owned patent applications identified above provide useful information in this regard.)

In one arrangement, profile data about an applicant is collected in an XML-based data structure, based on a collection of standardized tags. Part of an illustrative collection of data for an applicant may be as follows

<BIRTHDATE> 12/19/1945
<CURRENT_RES_ADDRESS> 6299 SW Tower Way, Portland, OR
97221
<PRIOR_RES_ADDRESS_1> 3609 SW Admiral St., Portland, OR
97221
<PRIOR_RES_ADDRESS_2> 5544 SW 152nd Ave., Portland, OR
97226
<HIGH_SCHOOL_1> Summit High School, Summit, NJ
<HIGH_SCHOOL_2> Wilson High School, Portland, OR
<COLLEGE_1> Washington State University, Pullman, WA
<COLLEGE_2> University of Waterloo, Ontario, Canada
<COLLEGE_STUDY_1> Geology
<OCCUPATION_1> Professor
<OCCUPATION_2> School of Earth Sciences
<OCCUPATION_3> University of Portland
<CITIZEN> USA

Such a collection of data can be seeded by information provided by the applicant, e.g., name, address, phone number, and social security number. The collection can then be supplemented by further information obtained from public sources (e.g., the web and public databases), as well as private data collections.

For example, credit agency databases (e.g., Experian, Equifax and Transunion) typically store prior addresses for myriad people, in addition to their current addresses. Based on address data, a wealth of additional information can be obtained. Various public web sites, for example, can provide corresponding provide latitude/longitude information. Online municipal tax databases can be queried to obtain information about the home (three bedrooms, two full baths, cedar shake roof, etc.). Third party commercial databases can provide statistical demographics about the neighborhood (e.g., average income, age distribution, percentage of renters, etc.). All of this information can be added to the profile (or accessed online, as needed).

Knowing alleged facts about the applicant allows the DMV to pose questions that help establish confidence in the applicant's asserted identity. The questions are generally of a sort that can be correctly answered by the applicant—if the facts collected concerning the applicant are correct, but would be more difficult to answer otherwise.

One knowledge domain with which every applicant should be familiar is facts about their residence. Given a residence address, online tools can be used to mine substantial information about the building, its construction, its features, its age, etc., as well as about the surrounding neighborhood.

Consider an applicant who asserts that his or her residence address is 6200 SW Tower Way, Portland, Oreg. By providing this address to the web page at portlandmaps.com or zillow.com, a wealth of information—parts of which are shown in FIGS. 1A and 1B, can be obtained. Questions can be posed to the applicant based on the facts provided from this web resource. For example

• • How many bathrooms does 6200 SW Tower Way have?
    • 1. One
    • 2. One and a half
    • 3. Two
    • 4. Two and a half
    • 5. Three
    • 6. Three and a half
  • When was 6200 SW Tower Way built?
    • 1. Prior to 1940
    • 2. Between 1940 and 1990
    • 3. Between 1991 and 2000
    • 4. After 2001
  • What type of heating system does 6200 SW Tower Way have?
    • 1. Forced air
    • 2. Radiant floor heat
    • 3. Baseboard hot water
      If the applicant identifies himself as the owner of the property (rather than a renter), then the applicant might be further asked:
  • In what year did you purchase the property?
    • 1. 1996
    • 2. 2000
    • 3. 2002
    • 4. 2004
    • 5. 2005
  • What was the purchase price for the property?
    • 1. Less than $180,000
    • 2. Between $180,000 and $220,000
    • 3. Between $220,001 and $260,000
    • 4. Between $260,001 and $300,000
    • 5. More than $300,000
      Each of these answers can be readily checked from the information available on-line, and depicted in FIGS. 1A and 1B.

If the applicant answers each of these questions correctly, it lends credence to their assertion that their residence is 6200 SW Tower Way. If the applicant answers most of the questions incorrectly, it raises a serious doubt as to at least their residence address. (Other information provided by such applicant may also be called into doubt.)

The web increasingly offers a rich trove of geospatial data, such as maps, aerial imagery, and curbside imagery, which is readily accessible by street addresses or other location data. The maps are of different types, e.g., street maps, zoning maps, topographic maps, utility (e.g., water, sewer, electrical) maps, hazard maps (e.g., flood plain, earthquake), etc. (The portlandmaps.com site referenced above includes all these categories of information.) Such resources offer a rich source of information about which an applicant can be questioned—based on current and/or prior residence addresses. Consider, e.g., the following questions which may be posed to an applicant who asserts his address is 6200 SW Tower Way, Portland, Oreg.:

• • Refer to the map shown in FIG. 2 (which includes your residence address) for the following questions.
    • What is the name of the park (“E”)?
      • 1. Albert Kelly Park
      • 2. April Hill Park
      • 3. Custer Park
      • 4. Dickinson Park
      • 5. Gabriel Park
      • 6. Hillsdale Park
      • 7. Pendleton Park
      • 8. None of the above
    • What is the name of the store located at the intersection of SW Vermont St and SW 45^th avenue?
      • 1. 7-11
      • 2. Express Mart
      • 3. Jackpot Food Mart
      • 4. Plaid Pantry
      • 5. Swan Mart
      • 6. Uptown Market
    • Is that store located at corner A, B or C?
      • 1. A
      • 2. B
      • 3. C
    • Is there a traffic light, or a four-way stop, at the intersection of SW Vermont St and SW 45^th avenue?
      • 1. Traffic light
      • 2. Four-way stop
    • Is location D uphill from location E?
      • 1. Yes
      • 2. No
    • Wilson High School is off the map. In what direction, from the map, is it?
      • 1. Off the top side
      • 2. Off the right side
      • 3. Off the bottom side
      • 4. Off the left side
        Again, an applicant who truly lives at 6200 SW Tower Way would have little difficulty with such questions. However, an imposter would fare poorly.

Typically, the map about which the applicant is questioned would be presented on an electronic display device, e.g., on a testing kiosk at a DMV office. Alternatively, the question could be posed to the applicant at a remote terminal (e.g., at the applicant's home)—provided certain safeguards are put in place to prevent the applicant from researching the answer. (E.g., the applicant would have a limited period of time after the presentation of each question to provide the answer.)

Similarly, consider the following questions, which may be posed to an applicant, and which proceed with reference to two aerial photographs:

• • Refer to the aerial photograph shown in FIG. 3 (which includes your residence address) for the following questions.
    • Which of the buildings is your residence (A-V)?
    • What is the last name of one of your neighbors?
    • In what building does that neighbor live (A-V)?
    • What is the name of the street labeled W?
      • 1. SW Tower Way
      • 2. SW Dakota St
      • 3. SW Idaho St
      • 4. SW Caldew Dr
    • If you drove the street labeled X from the top of the map towards the bottom, would you be going uphill or downhill?
      • 1. Uphill
      • 2. Downhill
  • FIG. 4 is an aerial photograph showing one of the intersections nearest your residence (about 0.4 miles away). In the map:
    • What business occupies building Y?
      • 1. Multnomah Community Center
      • 2. OHSU Clinic
      • 3. Big 5 Sports
      • 4. Tursi's Soccer Store
    • What business occupies building Z?
      • 1. Gerber Labs
      • 2. Marquam Gymnasium
      • 3. Southwest Community Center
      • 4. Whole Foods Market

It will be recognized that many such questions follow a standard form (e.g., a template) that can be recalled and customized by an automated process that inserts data unique to that particular neighborhood.

(In the examples given above, the maps/photographs used in some of the questions give away answers to other questions. Naturally, in practical application, this would be avoided.)

Source material for the foregoing queries can come from commercial providers (e.g., Google Earth, Terraserver.com, Yahoo!, MapQuest, etc.), or from government databases (national, state, municipal, etc.) The overlay of “A” “B” “C” markings, etc., can be provided—as desired—by the DMV or other entity.

Yet another source of information useful in quizzing applicants is Amazon's Yellow Pages search, and its A9 search engine. Amazon has compiled a library of curbside-scene images, accessible by street address or business name. Various public taxation districts also publish such information. Again, such imagery can be used to test an applicant's familiarity with the neighborhood in which she claims to reside.

To illustrate, imagine that the applicant gives an address that is between two town centers, e.g., Portland and Beaverton. A mapping database can be used to identify the fastest route between the stated address and one of the towns. A major intersection along this route (preferably near the applicant's residence) can be identified. A curbside imagery database (e.g., Amazon Yellow Pages A9) can then be queried to obtain a scene from this intersection. This image, or an unrelated image, can be presented to the applicant with a question such as:

• • Referring to FIG. 5, is this scene:
    • 1. Between your residence and Beaverton?
    • 2. Between your residence and Portland? or
    • 3. Not familiar to you.

Microsoft's Windows Live Local service (aka Virtual Earth) is yet another source of geographic image/map data against which the knowledge of an applicant may be tested. Microsoft's service offers “Bird's eye” (oblique), in addition to “aerial” (straight down), photographic imagery of many city locations. FIG. 6, for example, shows a “Bird's eye” view of part of the campus of the Georgia Institute of Technology, obtained through the Microsoft service. Such imagery can be used in the foregoing examples.

Another class of resources that might be tapped, in this example as in others, are online services provided by certain municipal bus services. Portland's “Tri-Met” transit system, for example, offers a web-based service (at the domain trimet.org) by which users can specify their start points and desired destinations, and the service identifies buses and trains, and their schedules, that the user can take to commute between the locations. The system's web services also offer local maps, showing which bus routes travel along which streets, and the location of bus stops. A user can specify an intersection, and obtain a map of bus routes near that intersection. Such a map, and the associated user interface, is shown in FIG. 7.

Such transit system information can be used to assess an applicant's knowledge about their alleged residence. An applicant may be asked, for example:

• • S.W. Dakota Street is near your home. Do mass transit buses run down that street?
    • 1. Yes.
    • 2. No.

The DMV may sometimes ask questions expecting that the applicant will not know the answer. For example, a person living in the neighborhood depicted in the bus map of FIG. 7 may be asked to name one or more bus routes that travel SW Beaverton-Hillsdale Highway (e.g., 54, 56, 61 and 92). If the applicant cannot answer this question, he is not faulted; the answer is likely unfamiliar to many neighborhood residents. If, however, the applicant can answer this difficult question correctly, such correct answer may help his score more than a correct answer to a more routine question. (The ability to identify two or more of the buses along this route could boost his score still further.) Thus, answers to different questions may be weighted differently in scoring confidence in the applicant's asserted identity.

Google offers a Transit Trip Planner service (google.com/transit) that seeks to standardize delivery of mass transit schedule and route information across many cities' transportation services. It, too, may be used as a fact resource in compiling challenge questions for applicants.

The neighborhood-based questions noted earlier are just a start. Others may include the following:

• • Leaving your home, what street intersection do you first encounter? What if you go the opposite way?
  • What cable TV service provider serves your house/neighborhood? (DSL provider? Electric utility company?)
  • What big mall is closest to your home? (Its anchor tenants?) What supermarket(s)?
  • Which town(s) adjoin the town where you live/work?
  • —What bus route is close to your home/work?
  • What is the name of the local mass transit system?
  • Does your side of your home street have a curb? A sidewalk?
  • Are power poles/lines on side of street where you live? (Or are they buried in your neighborhood?)
  • On what street is the Post Office in your community?
  • What school is near your home?
  • What intersection with traffic light is close to your home?
  • Name/number of thoroughfare that links town to adjoining town of [X]
  • On what day(s) does garbage get picked up from your home?
  • In what year did you purchase (or mortgage, or refinance) home? If not a homeowner, what is name of landlord/agency to whom you pay rent?
  • Name a person who lives next door.

The foregoing discussion has focused on a single knowledge domain—information that an applicant may be expected to know by reason of their residence address. Many other knowledge domains exist. One, for example, is information an applicant may be expected to know by reason of their employment. A few examples of facts on which an applicant may be quizzed, based on their employment, follow:

• • If a truck driver
    • Validity period of commercial driving license?
    • Meaning of acronym GVW?
    • Location of highway permit offices?
    • Location of highway weigh stations?
    • Weight restrictions on major roads/bridges?
    • Phone number of company's freight dispatcher?
    • Tolls on local roads?
  • If a lawyer
    • Names of local judges?
    • Web site of county bar association?
    • Knowledge of CLE requirements (e.g., how often must you report; how many hours/year required)?
    • Town where state bar association has its headquarters?

It will be recognized that many of these questions are not strictly based on employment, but also depend on location of employment. A truck driver in Portland, for example, may be able to name seven bridges in Portland, but none in Seattle. Generally speaking, the more closely tailored the questions are to the applicant profile (e.g., residence/employment/age/etc), the more useful they will be in establishing (or refuting) confidence in the asserted identity.

Another knowledge domain comprises facts an applicant may be expected to know by reason of their education. For example:

• • If a HS student or graduate
    • Name of principal of HS where/when graduated (or teacher)?
    • Size of HS graduating class (in tiered ranges)?
    • Sports team name/mascot?
  • If a college graduate
    • (Above questions for HS, adapted to college)
    • Town where college is located?
    • Zip code of that town?
    • Is school on schedule of 2 semester or 3 quarter/yr?
    • Name of a dorm on campus?
    • Name of a professor?
  • If degreed in pharmacology
    • Generic name for active ingredient in Benadryl?
    • Name of company that is successor to A.H. Robins?
    • City and/or state where Merck is headquartered?
    • Name of college where received pharmacology degree?
  • If a lawyer
    • Name of an early Supreme Court justice?
    • Meaning of force majeure?
    • Name 3 traditionally-required first year courses?

Some of this information may be readily mined from public databases. For example, the XML profile excerpted above specified that the applicant attended Wilson High School in Portland, Oreg. Querying the online database Google with the input “wilson high school portland mascot” gives, as the first search result, a “hit” from the online encyclopedia Wikipedia revealing that that the team mascot is the “Trojans.” See FIG. 8.

Some of this information may not appear as the first hit of a Google search, but can be researched with little effort. For example, information relating to high schools and colleges, including size and faculty (currently and historically) can be obtained by web sites maintained by the respective high schools and universities, and also from databases maintained by independent providers, such as classmates.com.

Some of this information can be garnered from independent human searchers, e.g., using Amazon's Mechanical Turk service. Amazon's Turk web site explains:

• • Amazon Mechanical Turk provides a web services API for computers to integrate Artificial Artificial Intelligence directly into their processing by making requests of humans. Developers use the Amazon Mechanical Turk web services API to submit tasks to the Amazon Mechanical Turk web site, approve completed tasks, and incorporate the answers into their software applications. To the application, the transaction looks very much like any remote procedure call—the application sends the request, and the service returns the results. In reality, a network of humans fuels this Artificial Intelligence by coming to the web site, searching for and completing tasks, and receiving payment for their work.
  • All software developers need to do is write normal code. The pseudo code below illustrates how simple this can be.

	read (photo);
	photoContainsHuman = callMechanicalTurk(photo);
	if (photoContainsHuman == TRUE){

acceptPhoto;

	}
	else {

rejectPhoto;

	}

More information about Amazon's Mechanical Turk service is provided in Appendix B (Amazon Mechanical Turk Developer Guide, 2006, 165 pp., API Version Oct. 31, 2006).

Alternatively, automated processes, e.g., using more traditional artificial intelligence techniques, can be applied to generate and check questions from available online resources, such as the maps and databases noted above, given profile data on the applicant (e.g., current and prior residence addresses, education, city where attended high school, etc.).

There are many other domains of knowledge about which an applicant can be questioned. One is facts the applicant may be expected to know by reason of who they know. For example, they may be asked to provide four memorized telephone numbers, and name the person to which each corresponds. Another, as noted earlier, is names of neighbors.

Still another knowledge domain comprises facts the applicant may be expected to know by reason of their age. For example:

• • Who was Nixon's VP?
  • In what state did Rosa Parks become famous?
  • What was Muhammad Ali's real name?
  • Who did Elizabeth Taylor many more than once?
  • Who did Brad Pitt recently break up with?
  • What was Dustin Hoffman's first famous role?
  • How many stars did the US flag have during WWII?
  • What animal is on the back of old nickels?

Yet another knowledge domain is facts the applicant may be expected to know by reason of activities they earlier performed. For example, readily accessible databases reveal answers to the following questions on which an applicant may be tested:

• • Did you cast a ballot in the November, 2004, election? How about the presidential race in 2000? In what county/state?
  • Did you donate money to the electoral campaign of any official? Who?
  • Have you had a motor vehicle violation in past 5 years? (In what state?)
  • Have you ever been summoned to jury duty?

While quizzing an applicant, some questions may be asked to compile data that can be a source for questions in future checking—either for the applicant himself/herself, or for another person. Examples include:

• • Name your first pet/your favorite teacher/your favorite frequent flier account number/your first telephone number/your earliest-remembered zip code.
  • Name a non-family member with whom you've lived (e.g., a college roommate); name a former neighbor; name a person who sits near you at work.
  • Name a person who was with you when you learned of the 9/11 attack, and where you were when you heard the news.

Another factor that can help confirm identity is an on-line account. Many individuals have accounts with on-line entities. Examples include Amazon, PayPal, EBay, Orbitz, Expedia, NetFlix, etc. Such accounts can provide verification information—either with the companies' cooperation, or without.

Consider Amazon. Its web site allows registered users to sign-in and display historical information, including a list of all orders ever placed with the company (at last back to 1997, as of this writing). For each order, Amazon provides information including the address to which it was shipped, the shipping date, and the billing address.

As part of a verification protocol, a DMV clerk may direct a web browser on a DMV computer to the Amazon Sign-In page. The applicant can then type his/her account information (e.g., email address and password), and navigate from the resulting “My Account” page to select “View by Item—all items ordered” (FIG. 9). A report like that shown in FIG. 10 is produced, listing all items ordered through that account. Clicking on an order number generates a screen like that shown in FIG. 11, showing both the address to which the item was shipped, as well as the billing address for the credit card used. If this address information is consistent with the address given by the applicant, this tends to confirm the applicant's credibility. In embodiments in which such factors influence an applicant's score, this would tend to increase that score. Naturally, the further back in time the applicant can demonstrate residence at the current address, the more the applicant's score might be increased. (Evidence that Amazon shipped a book to the stated address a month ago would be less assuring than evidence that Amazon shipped a book to that address a year, or five years, ago.)

Other on-line services can also be used for this purpose. EBay reports the date on which people joined and became members, and their location (e.g., “Member since Apr. 18, 1998; Location: United States”). Again, a demonstrable history with such an online vendor can be factored into the analysis of assessing whether the applicant is who they purport to be.

Moreover, EBay and other sites provide various mechanisms by which members are critiqued by their peers or others users. EBay thus reports a feedback score, indicating the percentage of unique users who have scored their transactions with a particular member as favorable. In one example, member “mgmill” has a feedback score of 100%: 2773 members left positive feedback; none left negative feedback. (A 100% score based on thousands of feedbacks is naturally more meaningful than a 100% score based on a dozen feedbacks.) Such peer/user rankings serve as an additional factor on which an applicant can be scored (albeit one that might be given relative little weight—absent some demonstrated correlation between EBay feedback scores, and the metric that the agency is trying to assess).

Expedia, Orbitz, and other online travel companies have histories that a user can access giving information useful in verification. From the Expedia home page, for example, a user can click on “My Account” and sign-in by entering a login name and password. An Account Overview page is then produced. This page lists the account holder and others for whom that person has made travel arrangements. Each name is hyperlinked to a further screen with information about that traveler, such as home phone number, work phone number, cell phone number, passport number and country, frequent flier numbers, etc. At the bottom of the screen there is a listing of credit cards associated with the account—including billing addresses and telephone numbers. Again, all such information can be accessed—with the cooperation of the applicant—and factored into a possibility-of-fraud fraud assessment for that applicant.

PayPal offers further data useful in checking a person. (PayPal currently has over 80 million accounts.) Again, going to the PayPal home page yields a screen on which the applicant can enter their email address and PayPal password. From the resulting “Overview” screen, the user can navigate to the “Profile” page. Here again, a great wealth of information useful in corroborating an applicant is available, e.g., links to email, street address, telephone, credit card accounts registered with PayPal, bank accounts registered with PayPal, etc. Clicking on the credit card account link produces a screen giving the last digits of the credit card number, and the credit card billing address. Clicking on the “History” tab allows historical transactions to be reviewed. Like Amazon, histories going back several years are available. As before, a long-tenured account may be taken as a favorable factor in assessing an applicant.

Again, for any particular PayPal transaction, further details can be retrieved (e.g., by clicking the “Details” link). These details include the address to which the purchased item was to be sent, and information about the credit card or bank account that was debited. Again, this information can be collected as part of the verification process, and checked for consistency with other information about the applicant.

By such techniques, ID verification can be augmented by allowing the applicant to demonstrate knowledge of accounts with which the applicant is associated. Knowledge of passwords, order history, and consistency between shipping addresses and applicant's asserted address, all tend to boost a confidence score.

Still richer data may be obtained by establishing a commercial relationship with companies having accounts with large numbers of consumers, so that additional information—not generally available—might also be made available.

(Naturally, safeguards may be put into place to assure privacy of the password and other data entered into the computer by the applicant to access such information, as well as of the account information thereby revealed.)

Such techniques serve to enlarge the “neighborhood” of those who can vouch for an individual, to encompass entities involved in commercial transactions.

Still another source from which question data can be derived is information provided by prior applicants. Applicants can be queried for information (questions and answers, or facts) that would most likely be known to other applicants having a similar background qualification, but would likely be unknown to others. Thus, a person who is a lawyer in Portland, Oreg., may offer the question, “Who is the old federal courthouse named after?” and offer the answer: “Gus Solomon.” Likewise, a bus driver in the same city may offer “Where is the bus garage on the west side of the Willamette River? Answer: Merlo Street.” A teacher at Hayhurst Elementary School in Portland may offer “To what middle school do Hayhurst students go? Answer: Robert Gray Middle School.”

A nineteen year old student at the University of Portland may offer, “What live music club is closest to school? Answer: Portsmouth Club.” A sixty year old faculty member at the same school may offer, “What is the Catholic order with which the University is affiliated? Answer: Holy Cross.”

It will be recognized that the nineteen year old student at the University of Portland may be less likely to know the answer to the latter question, and the sixty year old faculty member may be less likely to know the answer to the former question. Thus, in many cases, questions should not be selected for presentation to an applicant based on just a single factor (e.g., a connection to the University of Portland, or a residence on S.W. Dakota Street). Rather, the questions may be assigned based on two or more factors (e.g., age plus school affiliation; profession plus address; ethnicity plus location where grew up; etc.).

One way of conceptualizing this is as a multidimensional vector space in which both questions and applicants can be located.

The questions can each be located based on two or more classification data that help identify applicants who are likely to be able to answer such questions. Examples of possible classification data includes geographic location to which question relates, professional expertise (e.g., law) useful in answering question, age of persons most likely to be familiar with question, education of persons most likely to be familiar with questions, etc.

The applicant can be similarly located in this vector space, again based on factors such as noted above.

The questions nearest the applicant in this vector space are thus those that most closely match that applicant's background and other credentials, and are thus most likely to be useful in confirming that the applicant's background/credentials are as he or she states.

In cases where candidate questions are proposed by applicants, such questions can be tagged with classification data based on the proposer's attributes. E.g., if a Portland, Oreg. lawyer, aged 60, proposed the question about the name of the old courthouse, then that question might be stored with such applicant attributes (e.g., lawyer, Oregon, age 60) as classification data. (Classification data may be of the same type, or may even be drawn from the same vocabulary, as applicant attributes, but this need not be the case.)

In one arrangement, question data from which questions are drawn is stored in a large database, each record tagged with plural of the classification data noted earlier. Proximity between plural of the question data, and the particular applicant is then determined, so as to identify a subset of the original question space from which questions might usefully be posed. The method can then select questions from this subset (e.g., randomly) for presentation to the applicant.

It should be noted that question data need not be a single query to which there is a single answer. Question data can span a larger body of information, e.g., names of familiar Nobel laureates in physics from 1910-1935, and can provide fodder for many different, but related, questions.

In some arrangements, new question information collected from applicants can be vetted prior to use in verifying further applicants. Such vetting can be performed by DMV personnel, or by contractors (“professional vetting,” which may include the Mechanical Turk service). Computer verification may be employed in some instances.

Sometimes, the vetting may be performed by presenting such questions/answers to new applicants, on a trial basis. Answers to such questions would likely not be included in a substantive assessment as to the new applicant's identity or risk. Rather, such answers would be used to judge whether the question/answer is a useful tool.

For example, each proposed new question might be posed to five new applicants, each of whom appears to have a background that would allow them to correctly answer same. If at least three of them (or, alternatively, if at least four of them, or if all of them) give the answer offered by the proposer of the question, then the question can be moved to the pool of real questions—and used to substantively assess applicants.

(If the trial question is proposed to an applicant who does not know the answer, and other circumstances give rise to uncertainty whether such applicant's identity is bona fide or fraudulent, then such data point may be disregarded in assessing whether the question is a useful tool. Conversely, if the question is answered correctly by a person whose identity is otherwise established to be fraudulent, then this suggests that the question is not a useful discriminator tool.)

Combinations of professional and applicant vetting can also be used. For example, a candidate new question may be posed to five new applicants on a trial basis—as outlined above. If a sufficient number answer correctly, it can be further vetted by a professional. Or, such order may be reversed.

Tests of the sort detailed above might be posed routinely to applicants. If an applicant does not demonstrate knowledge of an expected sort, this can trigger further scrutiny of other aspects of the applicant's application. For example, a DMV agent might respond by more closely inspecting breeder documents presented by the dubious applicant, or by requesting additional information not required from routine applicants.

Alternatively, the checks presented above might be posed only if an applicant's profile otherwise suggests further checking is prudent (e.g., if a risk score based on the ensemble of presented breeder documents exceeds a threshold).

In most embodiments, an incorrect answer isn't fatal to applicant verification. There may be many credible explanations for an incorrect answer. Rather, answers are typically used to adjust an aggregate score—with no one question being determinative.

Applicants' technology detailed herein may be regarded as a knowledge-based system—utilizing a knowledge base of information to perform or facilitate an applicant-verification process. Earlier work in such knowledge-based systems is shown, e.g., by U.S. Pat. Nos. 6,968,328, 6,965,889 and 6,944,604.

As referenced above, the Mechanical Turk service can be widely used in gathering and vetting information used in the foregoing. (Additionally, the Mechanical Turk service can be used in conjunction with the systems detailed in the earlier-referenced patent documents to facilitate some of the operations required by those systems, e.g., making judgments and undertaking tasks that computers are ill-suited to perform—such as performing fuzzy matching, applying common-sense knowledge, and interpreting documents. Thus, for example, such a system understands that the names “Bill,” “Will,” “Wm.,” and the like, all can be acceptable matches to the name “William;” likewise that address lines “Apartment 3A,” “Apt. 3A,” “Unit 3A,” and the like, all can be acceptable matches to the address line “Apt. 3-A,” etc. Similarly, the Turk service can be used to harvest public records data that can be used in verification operations. A number of such applications of the Mechanical Turk service to the arrangements in the cited documents are within the capabilities of the artisan from the teachings herein. Appendix A further details some further inventive uses of the Mechanical Turk service, and similar “crowdsourcing” technologies.)

As noted, the arrangements detailed herein are not limited to verifying identity prior to issuance of identity documents; they have applicability in many other contexts.

To give but one example, consider a passport control checkpoint in an airport, where a government official inspects passports of travelers. When the passport is swiped or scanned by a passport reader, the official is presented with a screen of information pertaining to the person. This same screen, or another, can be used to present one or more verification checks like those detailed above, e.g., showing a map of the passport holder's neighborhood, and requesting the traveler to identify her home.

Naturally, in cases where the applicant data is received in advance of applicant testing, e.g., from a home web session, then additional time is available to prepare questions customized to that applicant.

No particular methodology for generating a score has been detailed above, because same depends on the facts of particular cases. One technique for generating a numeric score is a polynomial approach (such as that detailed in US20040153663), where different factors are weighted differently, and summed to produce an aggregate score. Scores may be produced based just on the information collected through the procedures detailed in this detailed description, or the scored can be based on a larger set of data, e.g., including factors on which scores are computed in the prior art.

Implementation of systems employing the foregoing principles is straightforward to artisans, e.g., using standard computer-, database-, software- and network-technology.

To provide a comprehensive disclosure without unduly lengthening this specification, applicants incorporate-by-reference the documents referenced in this disclosure (including Appendices). It is expressly contemplated that the technologies, features and analytical methods detailed herein can be incorporated into the methods/systems detailed in such other documents. Moreover, the technologies, features, and analytical methods detailed in those documents can be incorporated into the methods/systems detailed herein. (It will be recognized that the brief synopses of prior documents provided above naturally do not reflect all of the features found in such disclosures.)

In view of the wide variety of embodiments to which the principles and features discussed above can be applied, it should be apparent that the detailed embodiments are illustrative only and should not be taken as limiting the scope of the disclosed technology. Rather, we claim all such modifications as may come within the scope and spirit of the following claims and equivalents thereof.

APPENDIX A

Some Further Applications of Crowdsourcing Strategies

The Mechanical Turk service (detailed in Appendix B) may be regarded as a structured implementation of a technology commonly termed “crowdsourcing”—employing a group of outsiders to perform a task. Wikipedia explains:

• • “Crowdsourcing” is a neologism for a business model that depends on work being done outside the traditional company walls: while outsourcing is typically performed by lower paid professionals, crowdsourcing relies on a combination of volunteers and low-paid amateurs who use their spare time to create content, solve problems, or even do corporate R&D. The term was coined by Wired magazine writer Jeff Howe and editor Mark Robinson in June 2006.
  • Crowds targeted for crowdsourcing include garage scientists, amateur videographers, freelancers, photo enthusiasts, data companies, writers, smart mobs and the electronic herd.

Overview

• • While not a new idea, crowdsourcing is becoming mainstream. Open source projects are a form of crowdsourcing that has existed for years. People who may not know one another work together online to create complex software such as the Linux kernel, and the Firefox browser. In recent years internet technology has evolved to allow non-technical people to participate in online projects. Just as important, crowdsourcing presumes that a large number of enthusiasts can outperform a small group of experienced professionals.

Advantages

• • The main advantages of crowdsourcing is that innovative ideas can be explored at relatively little cost. Furthermore, it also helps reduce costs. For example if customers reject a particular design, it can easily be scrapped. Though disappointing, this is far less expensive than developing high volumes of a product that no one wants. Crowdsourcing is also related to terms like Collective Customer Commitment (CCC) and Mass Customisation. Collective Customer Commitment (CCC) involves integrating customers into innovation processes. It helps companies exploit a pool of talent and ideas and it also helps firms avoid product flops. Mass Customisation is somewhat similar to collective customer commitment; however, it also helps companies avoid making risky decisions about what components to prefabricate and thus avoids spending for products which may not be marketable later.

Types of Crowdsourced Work

• • Steve Jackson Games maintains a network of MIB (Men In Black), who perform secondary jobs (mostly product representation) in exchange for free product. They run publicly or semi-publicly announced play-tests of all their major books and game systems, in exchange for credit and product. They maintain an active user community online, and have done so since the days of BBSes.
  • Procter & Gamble employs more than 9000 scientists and researchers in corporate R&D and still have many problems they can't solve. They now post these on a website called InnoCentive, offering large cash rewards to more than 90,000 ‘solvers’ who make up this network of backyard scientists. P&G also works with NineSigma, YourEncore and Yet2.
  • Amazon Mechanical Turk co-ordinates the use of human intelligence to perform tasks which computers are unable to do.
  • YRUHRN used Amazon Mechanical Turk and other means of crowdsourcing to compile content for a book published just 30 days after the project was started.
  • iStockphoto is a website with over 22,000 amateur photographers who upload and distribute stock photographs. Because it does not have the same margins as a professional outfit like Getty Images it is able to sell photos for a low price. It was recently purchased by Getty Images.
  • Cambrian House applies a crowdsourcing model to identify and develop profitable software ideas. Using a simple voting model, they attempt to find sticky software ideas that can be developed using a combination of internal and crowdsourced skills and effort.
  • A Swarm of Angels is a project to utilize a swarm of subscribers (Angels) to help fund, make, contribute, and distribute, a £1 million feature film using the Internet and all digital technologies. It aims to recruit earlier development community members with the right expertise into paid project members, film crew, and production staff.
  • The Goldcorp Challenge is an example of how a traditional company in the mining industry used a crowdsource to identify likely veins of gold on its Red Lake Property. It was won by Fractal Graphics and Taylor-Wall and Associates of Australia but more importantly identified 110 drilling targets, 50% of which were new to the company.
  • CafePress and Zazzle, customized products marketplaces for consumers to create apparel, posters, cards, stamps, and other products.
  • Marketocracy, to isolating top stock market investors around the world in head to head competition so they can run real mutual funds around these soon-to-be-discovered investment super-stars.
  • Threadless, an internet-based clothing retailer that sells t-shirts which have been designed by and rated by its users.
  • Public Insight Journalism, A project at American Public Media to cover the news by tapping the collective and specific intelligence of the public. Gets the newsroom beyond the usual sources, uncovers unexpected expertise, stories and new angles.

EXTERNAL LINKS AND REFERENCES

• The Rise of Crowdsourcing, Wired June 2006.
• Crowdsourcing: Consumers as Creators, BusinessWeek July 2006.

One use of the Mechanical Turk service (or similar crowdsourcing engines) is in connection with computationally difficult tasks, such as identification of audio, video and imagery content. These tasks are sometimes addressed by so-called “fingerprint” technology, which seeks to generate a “robust hash” of content (e.g., distilling a digital file of the content down to perceptually relevant features), and then compare the thus-obtained fingerprint against a database of reference fingerprints computed from known pieces of content, to identify a “best” match. Such technology is detailed, e.g., in Haitsma, et al, “A Highly Robust Audio Fingerprinting System,” Proc. Intl Conf on Music Information Retrieval, 2002; Cano et al, “A Review of Audio Fingerprinting,” Journal of VLSI Signal Processing, 41, 271, 272, 2005; Kalker et al, “Robust Identification of Audio Using Watermarking and Fingerprinting,” in Multimedia Security Handbook, CRC Press, 2005, and in patent documents WO02/065782, US20060075237, US20050259819, and US20050141707.

A particular example of such technology is in facial recognition—matching an unknown face to a reference database of facial images. Again, each of the faces is distilled down to a characteristic set of features, and a match is sought between an unknown feature set, and feature sets corresponding to reference images. (The feature set may comprise eigenvectors or shape primitives.) Patent documents particularly concerned with such technology include US20020031253, U.S. Pat. No. 6,292,575, U.S. Pat. No. 6,301,370, U.S. Pat. No. 6,430,306, U.S. Pat. No. 6,466,695, and U.S. Pat. No. 6,563,950.

These are examples of technology that relies on “fuzzy” matching. The fingerprint derived from the unknown content often will not exactly match any of the reference fingerprints in the database. Thus, the database must be searched not just for the identical content fingerprint, but also for variants.

Expanding the search to include variants hugely complicates—and slows—the database search task. To make the search tractable, one approach is to prune the database—identifying excerpts thereof that are believed to be relatively likely to have a match, and limiting the search to those excerpts (or, similarly, identifying excerpts that are believed relatively unlikely to have a match, and not searching those excerpts).

The database search may locate several reference fingerprints that are similar to the fingerprint of the unknown content. The identification process then seeks to identify a “best” match, using various algorithms.

Such content identification systems can be improved by injecting a human into the process—by the Mechanical Turk service or similar systems.

In one particular arrangement, the content identification system makes an assessment of the results of its search, e.g., by a score. A score of 100 may correspond to a perfect match between the unknown fingerprint and a reference fingerprint. Lower scores may correspond to successively less correspondence. (At some lower score, S_x, (perhaps 60) the system may decide that there is no suitable match, and a “no-match” result is returned, with no identification made.)

Above some threshold score, S_y, (perhaps 70) the system may be sufficiently confident of the result that no human intervention is necessary. At scores below S_y, the system may make a call through the Mechnical Turk service for assistance.

The Mechanical Turk can be presented the unknown content (or an excerpt thereof), and some reference content, and asked to make a comparison. (The reference content may be stored in the fingerprint database, or may be readily obtainable through use of a link stored in the reference database.)

A single item of reference content can be provided for comparison with the unknown content, or several items of reference content can be provided. (Again, excerpts may be used instead of the complete content objects. Depending on the application, the content might be processed before sending to the crowdsource engine, e.g., removing metadata (such as personally identifiable information: name, driver license number, etc.) that is printed on, or conveyed with, the file.)

The requested comparison can take different forms. The service can be asked simply whether two items appear to match. Or it can be asked to identify the best of several possible matches (or indicate that none appears to match). Or it can be asked to give a relative match score (e.g., 0-100) between the unknown content and one or more items reference content.

In many embodiments, a query is referred to several different humans (e.g., 2-50) through the Mechanical Turk service, and the returned results are examined for consensus on a particular answer. In some queries (e.g., does Content A match Content B? Or is Content A a better match to Content C?), a “vote” may be taken. A threshold of consensus (e.g., 51%, 75%, 90%, 100%) may be required in order for the service response to be given weight in the final analysis. Likewise, in queries that ask the humans to provide a subjective score, the scores returned from plural such calls may be combined to yield a net result. (The high and/or low and/or outlier scores may be disregarded in computing the net result; weighting can sometimes be employed, as noted below.)

As suggested, the data returned from the Mechanical Turk calls may serve as a biasing factor, e.g., pushing an algorithmically determined output one way or another, to yield a final answer (e.g., a net score). Or the data returned from the Mechanical Turk calls may be treated as a definitive answer—with results from preceding processes disregarded.

Sometimes the database search may reveal several candidate matches, all with comparable scores (which may be above the threshold S_y). Again, one or more calls to the Mechanical Turk service may be invoked to decide which match is the best, from a subjective human standpoint.

Sometimes the Mechanical Turk service can be invoked even in situations where the original confidence score is below the threshold, S_x, which is normally taken as indicating “no match.” Thus, the service can be employed to effectively reduce this threshold—continuing to search for potential matches when the rote database search does not yield any results that appear reliable.

The service can also be invoked to effect database pruning. For example, a database may be organized with several partitions (physical or logical), each containing information of a different class. In a facial recognition database, the data may be segregated by subject gender (i.e., male facial portraits, female facial portraits), and/or by age (15-40, 30-65, 55 and higher—data may sometimes be indexed in two or more classifications), etc. In an image database, the data may be segregated by topical classification (e.g., portrait, sports, news, landscape). In an audio database, the data may be segregated by type (spoken word, music, other). Each classification, in turn, can be further segregated (e.g., “music” may be divided into classical, country, rock, other). And these can be further segregated (e.g., “rock” may be classified by genre, such as soft rock, hard rock, Southern rock; by artist, e.g., Beatles, Rolling Stones, etc).

A call to the Mechanical Turk can be made, passing the unknown content object (or an excerpt thereof) to a human reviewer, soliciting advice on classification. The human can indicate the apparent class to which the object belongs (e.g., is this a male or female face? Is this music classical, country, rock, or other?). Or, the human can indicate one or more classes to which the object does not belong.

With such human advice (which, again, may involve several human reviewers, with a voting or scoring arrangement), the system can focus the database search where a correct match—if any—is more likely to be found (or avoid searching in unproductive database excerpts). This focusing can be done at different times. In one scenario it is done after a rote search is completed, in which the search results yield matches below the desired confidence level of S_y. If the database search space is thereafter restricted by application of human judgment, the search can be conducted again in the limited search space. A more thorough search can be undertaken in the indicated subset(s) of the database. Since a smaller excerpt is being searched, a looser criteria for a “match” might be employed, since the likelihood of false-positive matches is diminished. Thus, for example, the desired confidence level S_y might be reduced from 70 to 65. Or the threshold S_x at which “no match” is concluded, may be reduced from 60 to 55. Alternatively, the focusing can be done before any rote searching is attempted.

The result of such a human-focused search may reveal one or more candidate matches. The Mechanical Turk service may be called a second time, to vet the candidate matches—in the manner discussed above. This is one of several cases in which it may be desirable to cascade Mechanical Turk calls—the subsequent calls benefiting from the former.

In the example just-given, the first Mechanical Turk call aids in pruning the database for subsequent search. The second call aids in assessing the results of that subsequent search. In other arrangements, Mechanical Turk calls of the same sort can be cascaded.

For example, the Mechanical Turk first may be called to identify audio as music/speech/other. A second call may identify music (identified per the first call) as classical/country/rock/other. A third call may identify rock (identified per the second call) as Beatles/Rolling Stones/etc. Here, again, by iterative calling of a crowdsourcing service, a subjective judgment can be made that would be very difficult to achieve otherwise.

In some arrangements, human reviewers are pre-qualified as knowledgeable in a specific domain (e.g., relatively expert in recognizing Beatles music). This qualification can be established by an online examination, which reviewers are invited to take to enable them to take on specific tasks (often at an increased rate of pay). Some queries may be routed only to individuals that are pre-qualified in a particular knowledge domain. In the cascaded example just given, for example, the third call might be routed to one or more users with demonstrated expertise with the Beatles (and, optionally, to one or more users with demonstrated expertise with the Rolling Stones, etc). A positive identification of the unknown content as sounding like the Beatles would be given more relative weight if coming from a human qualified in this knowledge domain. (Such weighting may be taken into account when aggregating results from plural human reviewers. For example, consider an unknown audio clip sent to six reviewers, two with expertise in the Beatles, two with expertise in the Rolling Stones, and two with expertise in the Grateful Dead. Assume the Beatles experts identify it as Beatles music, the Rolling Stones experts identify it as Grateful Dead music, and the Grateful Dead experts identify it as Rolling Stones music. Despite the fact that there are tie votes, and despite the fact that no selection earned a majority of the votes, the content identification service that made these calls and is provided with these results may logically conclude that the music is Beatles.)

Calls to the Mechanical Turk service may request the human to provide metadata relevant to any content reviewed. This can include supposed artist(s), genre, title, subject, date, etc. This information (which may be ancillary to a main request, or may comprise the entirety of the request) can be entered into a database. For example, it can be entered into a fingerprint database—in association with the content reviewed by the human.

Desirably, data gleaned from Mechanical Turk calls are entered into the database, and employed to enrich its data—and enrich information that can be later mined from the database. For example, if unknown content X has a fingerprint F_x, and through the Mechanical Turk service it is determined that this content is a match to reference content Y, with fingerprint F_y, then a corresponding notation can be added to the database, so that a later query on fingerprint F_x (or close variants thereof) will indicate a match to content Y. (E.g., a lookup table initially indexed with a hash of the fingerprint F_x will point to the database record for content Y.)

Calls to outsourcing engines involve a time lag before results are returned. The calling system can generally cope, or be adapted to cope, with such lags.

Consider a user social networking site such as YouTube (now Google) that distributes “user generated content” (e.g., video files), and employs fingerprinting to recognize media content that should not be distributed. The site may check a video file at the time of its uploading with a fingerprint recognition system (e.g., of the sort offered by Audible Magic, or Gracenote). If no clear match is identified, the video may be indexed and stored on YouTube's servers, available for public downloading. Meanwhile, the content can be queued for review by one our more crowdsource reviewers. They may recognize it as a clip from the old TV sitcom “I Love Lucy”—perhaps digitally rotated 3 degrees to avoid fingerprint detection. This tentative identification is returned to YouTube from the API call. YouTube can check the returning metadata against a title list of works that should not be distributed (e.g., per the request of copyright owners), and may discover that “I Love Lucy” clips should not be distributed. It can then remove the content from public distribution. (This generally follows a double-check of the identification by a YouTube employee.) Additionally, the fingerprint database can be updated with the fingerprint of the rotated version of the I Love Lucy, allowing it to be immediately recognized the next time it is encountered.

If the content is already being delivered to a user at the moment the determination is made (i.e., the determination that the content should not be distributed publicly), then the delivery can be interrupted. An explanatory message can be provided to the user (e.g., a splash screen presented at the interruption point in the video).

Rotating a video by a few degrees is one of several hacks that can defeat fingerprint identification. (It is axiomatic that introduction of any new content protection technology draws hacker scrutiny. Familiar examples include attacks against Macrovision protection for VHS tapes, and against CSS protection for packaged DVD discs.) If fingerprinting is employed in content protection applications, such as in social networking sites (as outlined above) or peer-to-peer networks, its vulnerability to attack will eventually be determined and exploited.

Each fingerprinting algorithm has particular weaknesses that can be exploited by hackers to defeat same. An example will help illustrate.

A well known fingerprinting algorithm operates by repeatedly analyzing the frequency content of a short excerpt of an audio track (e.g., 0.4 seconds). The method determines the relative energy of this excerpt within 33 narrow frequency bands that logarithmically span the range 300 Hz-2000 Hz. A corresponding 32-bit identifier is then generated from the resulting data. In particular, a frequency band corresponds to a data bit “1” if its energy level is larger than that of the band above, and a “0” if its energy level is lower. (A more complex arrangement can also take variations over time into account, outputting a “1” only if the immediately preceding excerpt also met the same test, i.e., having a band energy greater than the band above.)

Such a 32 bit identifier is computed every hundredth of a second or so, for the immediately preceding 0.4 second excerpt of the audio track, resulting in a large number of “fingerprints.” This series of characteristic fingerprints can be stored in a database entry associated with the track, or only a subset may be stored (e.g., every fourth fingerprint).

When an unknown track is encountered, the same calculation process is repeated. The resulting set of data is then compared against data earlier stored in the database to try and identify a match. (As noted, various strategies can be employed to speed the search over a brute-force search technique, which yields unacceptable search times.)

While the just-described technique is designed for audio identification, a similar arrangement can be used for video. Instead of energies in audio subbands, the algorithm can use average luminances of blocks into which the image is divided as the key perceptual features. Again, a fingerprint can be defined by determining whether the luminance in each block is larger or smaller than the luminance of the preceding block.

While little has been written about attacks targeting fingerprinting systems, a casual examination of possible attack scenarios reveals several possibilities. A true hacker will probably see many more. Four simple approaches are discussed below.

Radio Loudness Profiling

The reader may be familiar with different loudness profiles selectable on car radios, e.g., Jazz, Talk, Rock, etc. Each applies a different frequency equalization profile to the audio, e.g., making bass notes louder if the Rock setting is selected, and quieter if the Talk setting is selected, etc. The difference is often quite audible when switching between different settings.

However, if the radio is simply turned on and tuned to different stations, the listener is generally unaware of which loudness profile is being employed. That is, without the ability to switch between different profiles, the frequency equalization imposed by a particular loudness profile is typically not noticed by a listener. The different loudness profiles, however, yield different fingerprints.

For example, in the Rock setting, the 300 Hz energy in a particular 0.4 second excerpt may be greater than the 318 Hz energy. However, in the Talk setting, the situation may be reversed. This change prompts a change in the leading bit of the fingerprint.

In practice, an attacker would probably apply loudness profiles more complex than those commonly available in car radios—increasing and decreasing the loudness at many different frequency bands (e.g., 32 different frequency bands). Significantly different fingerprints may thus be produced. Moreover, the loudness profile could change with time—further distancing the resulting fingerprint from the reference values stored in a database.

Multiband Compression

Another process readily available to attackers is audio multiband compression, a form of processing that is commonly employed by broadcasters to increase the apparent loudness of their signal (most especially commercials). Such tools operate by reducing the dynamic range of a soundtrack—increasing the loudness of quiet passages on a band-by-band basis, to thereby achieve a higher average signal level. Again, this processing of the audio changes its fingerprint, yet is generally not objectionable to the listeners.

Psychoacoustic Processing

The two examples given above are informal attacks—common signal processing techniques that yield, as side-effects, changes in audio fingerprints. Formal attacks—signal processing techniques that are optimized for purposes of changing fingerprints—are numerous.

Some formal attacks are based on psychoacoustic masking. This is the phenomena by which, e.g., a loud sound at one instant (e.g., a drum beat) obscures a listener's ability to perceive a quieter sound at a later instant. Or the phenomena by which a loud sound at one frequency (e.g., 338 Hz) obscures a listener's ability to perceive a quieter sound at a nearby frequency (e.g., 358 Hz) at the same instant. Research in this field goes back decades. (Modern watermarking software employs psychoacoustic masking in an advantageous way, to help hide extra data in audio and video content.)

Hacking software, of course, can likewise examine a song's characteristics and identify the psychoacoustic masking opportunities it presents. Such software can then automatically make slight alterations in the song's frequency components in a way that a listener won't be able to note, yet in a way that will produce a different series of characteristic fingerprints. The processed song will be audibly indistinguishable from the original, but will not “match” any series of fingerprints in the database.

Threshold Biasing

Another formal attack targets fingerprint bit determinations that are near a threshold, and slightly adjusts the signal to swing the outcome the other way. Consider an audio excerpt that has the following respective energy levels (on a scale of 0-99), in the frequency bands indicated:

	300 Hz	318 Hz	338 Hz	358 Hz
	69	71	70	68

The algorithm detailed above would generate a fingerprint of {011 . . . } from this data (i.e., 69 is less than 71, so the first bit is ‘0’; 71 is greater than 70, so the second bit is ‘1’; 70 is greater than 68, so the third bit is ‘1’).

Seeing that the energy levels are somewhat close, an attacker tool could slightly adjust the signal's spectral composition, so that the relative energy levels are as follows:

	300 Hz	318 Hz	338 Hz	358 Hz
	[69] 70	[71] 69	70	68

Instead of {011 . . . }, the fingerprint is now {101 . . . }. Two of the three illustrated fingerprint bits have been changed. Yet the change to the audio excerpt is essentially inaudible.

Exploiting Database Pruning

Other fingerprint hacking vulnerabilities arise from shortcuts employed in the database searching strategy—seeking to prune large segments of the data from further searching. For example, the system outlined above confines the large potential search space by assuming that there exists a 32 bit excerpt of the unknown song fingerprint that exactly matches (or matches with only one bit error) a 32 bit excerpt of fingerprint data in the reference database. The system looks at successive 32 bit excerpts from the unknown song fingerprint, and identifies all database fingerprints that include an excerpt presenting a very close match (i.e., 0 or 1 errors). A list of candidate song fingerprints is thereby identified that can be further checked to determine if any meets the looser match criteria generally used. (To allow non-exact fingerprint matches, the system generally allows up to 2047 bit errors in every 8192 bit block of fingerprint data.)

The evident problem is: what if the correct “match” in the database has no 32 bit excerpt that corresponds—with just 1 or 0 bit errors—to a 32 bit excerpt from the unknown song? Such a correct match will never be found—it gets screened out at the outset.

A hacker familiar with the system's principles will see that everything hinges on the assumption that a 32 bit string of fingerprint data will identically match (or match with only one bit error) a corresponding string in the reference database. Since these 32 bits are based on the strengths of 32 narrow frequency bands between 300 Hz and 2000 Hz, the spectrum of the content can readily be tweaked to violate this assumption, forcing a false-negative error. (E.g., notching out two of these narrow bands will force four bits of every 32 to a known state: two will go to zero—since these bands are lower in amplitude than the preceding bands, and two will go to one—since the following bands are higher in amplitude that these preceding, notched, bands). On average, half of these forced bits will be “wrong” (compared to the untweaked music), leading to two bit errors—violating the assumption on which database pruning is based.)

Attacks like the foregoing require a bit of effort. However, once an attacker makes the effort, the resulting hack can be spread quickly and widely.

The exemplary fingerprinting technique noted above (which is understood to be the basis for Gracenote's commercial implementation, MusicID, built from technology licensed from Philips) is not unique in being vulnerable to various attacks. All fingerprinting techniques (including the recently announced MediaHedge, as well as CopySense and RepliCheck) are similarly believed to have vulnerabilities that can be exploited by hackers. (A quandary for potential adopters is that susceptibility of different techniques to different attacks has not been a focus of academic attention.)

It will be recognized that crowdsourcing can help mitigate the vulnerabilities and uncertainties that are inherent in fingerprinting systems. Despite a “no-match” returned from the fingerprint-based content identification system (based on its rote search of the database for a fingerprint that matches that of the altered content), the techniques detailed herein allow human judgment to take a “second look.” Such techniques can identify content that has been altered to avoid its correct identification by fingerprint techniques. (Again, once such identification is made, corresponding information is desirably entered into the database to facilitate identification of the altered content next time.)

It will be recognized that the “crowdsourcing” methodologies detailed above also have applicability to other tasks involved in the arrangements detailed in the specification, including all the documents incorporated by reference.