REMOVING DUPLICATES BY TEXT SIMILARITY
Introduction to Removing Duplicates
To remove duplicates from a single column using Flookup, go to Extensions > Flookup Data Wrangler > Remove duplicates in your spreadsheet menu.
Removing Duplicates by Percentage or Sound Similarity
Click the menu item labelled "By percentage" or "By sound", depending on what you want to do.
Select the mode you want this function to run via the first drop-down menu. Your choices are:
Keep first unique value
Keep last unique value
Select text entries of one or more columns in your spreadsheet. Please note that if you select range A2:D500, for example, and duplicates are identified on row C10, C20 and C50, then A10:D10, A20:D20 and A50:D50 will be removed.
Click "Map columns in selection" in order to index the current columns in your selection.
Enter the Left_column index. If no user input is made, then the first column of the selected range will be analysed.
This step is only when running "By percentage". Enter the Threshold value. If no user input is made, only exact matches will be deleted.
Click "Remove duplicates".
Key Points on Removing Duplicates
Threshold values must increase or reduce by magnitudes of 0.05.
The Left_column value is the only column that will be analysed in this mode. This can be any integer representing a single column inside your selection.
If you are running "By percentage", then duplicates will be text entries within Left_column that have a level of similarity that is higher than or equal to the Threshold value.
If this function finishes running or times out, a message will be displayed indicating how many rows have been processed up to that point.
How to Remove Duplicates Across Two Different Columns
Click the menu item labelled "By percentage" or "By sound", depending on what you want to do.
Select the mode you want this function to run via the first drop-down menu i.e.
Keep first unique value
Keep last unique value
Select the second drop-down option labelled Compare two different columns.
Select text entries of two or more columns in your spreadsheet. This also defines number of columns that will be deleted for each row that contains duplicates.
Click "Map columns in selection" in order to index the current columns in your selection. Do this also if "Right_column" is inactive.
Specify your Left_column and Right_column index. These are the two columns that will be compared to each other.
If you are running the "By percentage" function, adjust the Threshold value to match your needs. Otherwise, skip this step.
Click "Remove duplicates".
Key Points on Removing Duplicates Across Different Columns
Before using this mode, it is advisable to first remove any duplicates within Left-column.
Threshold values must increase or reduce by magnitudes of 0.05.
Duplicates are values in Left_column that exist in Right_column and any row with a duplicate will be deleted.
Duplicates are text entries that have a level of similarity that is higher than or equal to Threshold between them.
If this function finishes running or times out, a message will be displayed indicating how many rows have been processed up to that point.
How to Remove Duplicate Rows
To remove duplicate rows by considering all columns within the selection e.g. comparing A2:E2 to A3:E3 for similarity, simply follow these steps:
Click the menu item labelled "By percentage" or "By sound", depending on what you want to do.
Select the mode you want this function to run via the first drop-down menu i.e.
Keep first identified duplicate value
Keep last identified duplicate value
Select the second drop-down option labelled Compare data in selection by row.
Select the data range of two or more columns to be analysed for duplicates. This also defines number of columns that will be deleted for each row that is considered a duplicate.
Click "Map columns in selection" in order to index the current columns in your selection.
If you are running "By percentage", adjust the Threshold value to match your needs. Otherwise, skip this step.
Click "Remove duplicates".
How to Roll Up Data in Duplicate Rows
To combine or concatenate duplicate rows by considering all columns within the selection e.g. combining A2:E2 , A3:E3 and A4:E4, simply follow these steps:
Click the menu item labelled "By percentage", depending on what you want to do.
Select the second drop-down option labelled Roll up data in selection by row.
Select the data range of two or more columns to be analysed for duplicates.
Click "Map columns in selection" in order to index the current columns in your selection.
If you are running "By percentage", adjust the Threshold value to match your needs. Otherwise, skip this step.
Click "Remove duplicates".
ULIST
Head to Extensions > Flookup Data Wrangler > Match and Merge Functions > ULIST in your spreadsheet menu.
Select the mode you want this function to run. Your choices are:
Keep first identified duplicate value
Keep last identified duplicate value
Primary range: Select range of one or more columns and click "Grab selected range".
Lookup_column: Enter the index of the column of values in "Primary range" that you want to analyse. If no user input is made, then the leftmost column of "Primary range" will be analysed.
Threshold: Enter the minimum percentage similarity. If no user input is made, then values that have a 0.6 level of similarity will be marked as duplicates and removed.
Click an empty cell in a column where you want your results to be displayed.
Click "Get unique values".
Key Points on ULIST
Threshold values must increase or reduce by magnitudes of 0.05.
If ULIST completes or times out, the results that have been processed up to that point, will be displayed.