FUZZY MATCHING ALGORITHMS EXPLAINED

What is Fuzzy Matching?

Fuzzy matching is a technique of finding strings in a dataset, that approximately match strings in a separate dataset, rather than exactly. The discipline of fuzzy matching can be typically sub-divided into two problems: 

Fuzzy matching is known by several names including fuzzy string matching and approximate string matching. Most fuzzy matching algorithms return similarity scores as percentages to help users gauge how similar the compared text entries are, with a typical scale ranging from 0% for no matches to 100% for exact matches.

Why Use Fuzzy Matching Software?

Real-world data is rarely stored in standardised formats because of the different methods of collecting and processing the same.

This usually leads to differences in spelling, formatting and other common data entry inconsistencies. Despite these text-based differences, the process of data cleaning can be vastly improved by making use of fuzzy matching software.

Fuzzy matching algorithms have been successfully applied in areas like nucleotide sequence matching, spell checking, spam filtering and record linkage.

To find out how to make use of these and other advantages for your personal or business data cleaning projects, please check out this article by Data Ladder.

Fuzzy Matching in Action: A Real-World Example

Record linkage techniques can be used to detect fraud, resource wastage or abuse. In this example, two databases were merged and compared for inconsistencies, leading to a discovery that helped the U.S. government put a stop to fraudulent behaviour by some government employees:

In a period of 18 months leading to the summer of 2005, a database comprising records of 40,000 pilots licensed by the U.S. Federal Aviation Administration and residing in Northern California, was matched to a database consisting of individuals receiving disability payments from the Social Security Administration, and it was discovered that names of some pilots appeared in both databases.

In a report by the Associated Press, a prosecutor from the U.S. Attorney’s Office in Fresno, CA stated the following:

There was probably criminal wrongdoing. The pilots were either lying to the FAA or wrongfully receiving benefits. The pilots claimed to be medically fit to fly airplanes. However, they may have been flying with debilitating illnesses that should have kept them grounded, ranging from schizophrenia and bipolar disorder to drug and alcohol addiction and heart conditions.

In the end, at least 40 pilots were charged with "Making false statements to a government agency" and "Making and delivering a false official writing". The FAA also suspended licenses of 14 pilots in total, while others were put on notice pending further investigations.

Popular Fuzzy Matching Algorithms