bruce perry, salesforce admin/developer
TRANSCRIPT
Manchester, NH 2019@NEDreamin #NEDreamin
With special thanks to our Platinum sponsors
Dealing With DuplicatesBruce Perry, Salesforce
Admin/Developer
Manchester, NH 2019@NEDreamin #NEDreamin
Manchester, NH 2019@NEDreamin #NEDreamin
Quiz Time
If you don’t have any duplicates,
A. Maybe you have a user adoption problem?B. Org-wide defaults set to Read-Only?C. You’re the only one with access to the database?D. Or maybe you’re just a really, really good admin?
Manchester, NH 2019@NEDreamin #NEDreamin
Duplicates are a data quality issue
They contribute to
● Inaccurate reports/estimates● Increased costs (time wasted, duplicated calls and mailings)● Makes your organization seem disorganized to customers/clients● User dissatisfaction
Bottom line: Duplicates can make Salesforce (and you) look bad.
Manchester, NH 2019@NEDreamin #NEDreamin
Where do duplicates come from?
● Users○ Lack of training (Search before you click new!)○ Bad processes
● Bulk data loads○ Import Wizard will check for duplicates(can load up to 50,000 records)○ Data Loader won’t check for duplicates○ Overlapping data sets○ Inaccuracy/inconsistency in initial data set
● Integrations/External apps that communicate with Salesforce● Attempts by alien civilizations to communicate with us via popular cloud apps
Manchester, NH 2019@NEDreamin #NEDreamin
Best Practice Alert!
Make a backup before you start - and put it somewhere safe.
Options:
● Back up everything (Salesforce export)● Use a report* to export a more targeted selection of fields/records● 3rd party backup tool (if you happen to have one available)
*If you use a report for this purpose, do not forget to include any relevant IDs (eg. account ID if backing up contacts)
Manchester, NH 2019@NEDreamin #NEDreamin
Best Practice Alert #2!
We interrupt the previous interruption to inform you that it is a best practice to have a unique identifier* for every batch of data that you upload.
It is A LOT easier to remove a batch of uploaded data if all members share a common key.
Deleting a batch of uploaded data is a very fast way to get rid of duplicates.
*doesn’t need to be fancy, your initials, the date, and maybe a number is enough.
Manchester, NH 2019@NEDreamin #NEDreamin
Where do we find duplicates?
● Contacts● Leads● Accounts (and sub accounts and household accounts)● Affiliations● Custom objects
Manchester, NH 2019@NEDreamin #NEDreamin
Steps to a duplicate-free database
1. Train your users2. Turn on duplicate detection/warning/reporting (reporting is a must)3. Develop reports on duplicates4. Make a backup!5. Merge critical duplicates manually6. Develop exact match scenarios.7. Merge duplicates from the top of your object hierarchy down.8. Develop fuzzy match scenarios.9. Merge duplicates from the top of your object hierarchy down.
Manchester, NH 2019@NEDreamin #NEDreamin
Preventing (or flagging) new duplicates
● Activate a standard matching rule● Or Create and Activate a custom matching rule● Activate standard duplicate rule● Or Create and Activate a custom duplicate rule
These are located under Setup
● Data○ Duplicate management
■ Matching rules
Manchester, NH 2019@NEDreamin #NEDreamin
Matching Rules and Duplicate Rules
Matching Rules (Standard or custom) tell Salesforce what qualifies as a duplicate.
Duplicate Rules tell Salesforce to handle any duplicates that are found.
Pro Tip - Don’t forget!!!Both types of rules need to be turned on.
If you turn on a matching rule without turning on its duplicate rule, Salesforce will do the comparisons, but take no action. Ask me how I know this.
Manchester, NH 2019@NEDreamin #NEDreamin
Matching Rules, part 1
Matching rules can get complicated*, You can have:
● Exact matching● Up to 10 conditions● Filter logic (AND/OR)● Fuzzy matching (multiple flavors!)
Before getting into the weeds, ask yourself, “What combination of matching fields in common use gives me a 95% chance of finding a duplicate?”
*Jaro-Winkler/Metaphone 3/Kullback Liebler complicated!
Manchester, NH 2019@NEDreamin #NEDreamin
Manchester, NH 2019@NEDreamin #NEDreamin
Fuzzy Matching
What fields have the biggest potential benefit from fuzzy matching?
● Phone number - (123) 456 7890 is the same as 123-456-7890● First name - Bob/Rob/Robert might be the same person
Also, possibly email, but Salesforce doesn’t have this option. BTW, did you know that
[email protected] = [email protected]
It’s true!
Manchester, NH 2019@NEDreamin #NEDreamin
Duplicate Rules, Part 1
Duplicate rules are simpler than matching rules. The major decision is to allow or block the duplicate. You can do that both when a record is being created AND when it is being edited. You can also alert the user and flag the duplicate for reporting purposes.
You can choose to enforce sharing rules - or not. If you’re getting duplicates because users can’t see records they are duplicating, consider turning this off.
You also get the option to add filter logic. You could have a duplicate rule that applied to certain account names only or contacts created after a certain date.
Manchester, NH 2019@NEDreamin #NEDreamin
New Flash!!!
If you want to do cross-object comparisons, the place you do them is the
Duplicate Rule
Manchester, NH 2019@NEDreamin #NEDreamin
Manchester, NH 2019@NEDreamin #NEDreamin
Turn on Duplicate Management gradually
You can turn duplicate reporting on and nobody will even know you’re doing it.
You can turn on both warn and report. Users can still work as they did previously, but they will be told when they are creating a potential duplicate.
Manchester, NH 2019@NEDreamin #NEDreamin
How are duplicates tracked?
Short version: background objects
Why is this useful to know?
Because you can use these objects for reports!
● Salesforce uses Duplicate Record Sets/Duplicate Record Items● DupeBlocker uses DupeBlocker Potential Duplicate● Duplicate Check uses Duplicate Check Group● Ringlead Cleanse does not keep info on matches in Salesforce
Manchester, NH 2019@NEDreamin #NEDreamin
Report Types for Duplicate Reporting
Generic duplicate reporting - across all object types
● Duplicate Record Set○ Duplicate Record Item
Specific object duplicate reporting:
● <Object Name>○ Duplicate record Item
Manchester, NH 2019@NEDreamin #NEDreamin
Report tricks for duplicate reporting, Part 1
Use a row level formula* to build a key field composed of multiple fields that you want to compare. You can then sort by the key field and see important values all in one place.
Example - <Full Name>-<City>-<State>-<Zip Code>
Maybe someday we’ll be able to group by row level formulas to make duplicates really stand out here. Not all fields available (eg. email and phone)
No added fields in your objects!
*you may need to enable these in your org first.
Manchester, NH 2019@NEDreamin #NEDreamin
Report Tricks, Example 1
Manchester, NH 2019@NEDreamin #NEDreamin
Report tricks for duplicate reporting, Part 2
Group by* columns you don’t usually use for groups (eg. email or cell phone)
The records that end up in groups might be duplicates
*Sort by works too, but Group by makes the matches stand out more
Manchester, NH 2019@NEDreamin #NEDreamin
Report Tricks, Example 2
Manchester, NH 2019@NEDreamin #NEDreamin
Report tricks for duplicate reporting, Part 2
Group by* columns you don’t usually use for groups (eg. email or cell phone)
The records that end up in groups might be duplicates
*Sort by works too, but Group by makes the matches stand out more
Manchester, NH 2019@NEDreamin #NEDreamin
Merging individual duplicates, Part 1
Classic
● Account - Accounts Tab | Tools | Merge Accounts● Contact - Individual Account | Contacts Section | Merge Contacts
In Salesforce Classic, accounts must have similar names to be merged
Lightning
● Account - Potential duplicates shown on account record● Contact - Potential duplicates shown on contact record
Manchester, NH 2019@NEDreamin #NEDreamin
Merging individual duplicates, Part 2
Lightning Limitation Alert!In Lightning, the Potential Duplicates component is only available for accounts, contacts, and leads. For any other object, you get the warning that you’re about to create a duplicate. After that warning, you have to find the duplicate record yourself.
Potential duplicates are probably based on your current rules. There’s no “Search” feature as in Classic.
Manchester, NH 2019@NEDreamin #NEDreamin
Merging individual duplicates, Part 3
NPSP
● Account (also Household) - Potential Duplicates pane on account (or switch to classic)
● Contact - App Menu | Merge Contacts
Remember, NPSP uses household accounts and relates contacts to accounts via primary affiliation, not a direct lookup. Households can be split and merged.
Contacts that are not on the same account can be merged (unlike Classic).
Manchester, NH 2019@NEDreamin #NEDreamin
Merging individual duplicates, Part 4
Merge higher level objects first, then related lower level objects. Merging the higher level objects may mean there are now duplicated lower level objects
For example, first Accounts, then Contacts.
Manchester, NH 2019@NEDreamin #NEDreamin
Before you merge, know your data
Are two contacts with these characteristics duplicates?
● same address, ● same birthday, ● same home phone, ● same last name, ● very similar first names
Manchester, NH 2019@NEDreamin #NEDreamin
Not necessarily
Meet the twins!
Frank and Frankie Smith**real world example, last name changed
Do you have a data guru? Find one before you start to de-duplicate.
Manchester, NH 2019@NEDreamin #NEDreamin
Manchester, NH 2019@NEDreamin #NEDreamin
Quiz Time!
Are you done with duplicates after merging two accounts and any resulting contact duplicates?
Manchester, NH 2019@NEDreamin #NEDreamin
Not necessarily
You might be using NPSP and might have duplicated affiliations
You might have custom objects related to the contact. These can become duplicates after merging contacts.
Manchester, NH 2019@NEDreamin #NEDreamin
Before Account and Contact Merge
Account
Contact
Affiliation
After
Account
Contact
Affiliation
Account
Contact
AffiliationAffiliation
Manchester, NH 2019@NEDreamin #NEDreamin
Finding and handling existing duplicates
● Reports on flagged duplicates (report type)● Basic reports● Salesforce Merge tools - Salesforce account and
contact merges, NPSP merges● Third party apps (next slide)
Manchester, NH 2019@NEDreamin #NEDreamin
Spotting/Fixing Duplicates in Bulk
Salesforce duplicate jobs (Performance and unlimited Editions only)
Third party tools:
● Duplicate Check● Cloudingo● Dupe Blocker● Demand Tools● Ringlead Cleanse● CleanCRM● ZaapIT DeDupe Manager
Manchester, NH 2019@NEDreamin #NEDreamin
Budget Alert!
It has come to my attention that some of these tools cost money. If getting money for these kinds of add-in tools is an issue for you, keep track of how long it takes to correct bulk duplicates by:
● Exporting data● Matching the duplicates in Excel● Deciding which fields to keep● Uploading updated data to the correct record● Deleting the duplicate record
The cost of these tools should be easy to justify in terms of your time saved (for cool new projects!)
Manchester, NH 2019@NEDreamin #NEDreamin
Spotting/Fixing Duplicates in Bulk
Don’t forget the backup!
Manchester, NH 2019@NEDreamin #NEDreamin
Required Features for Bulk Duplicate Handling
● Can save your search/scenario/job for re-use● Fuzzy matching of at least first name and phone● Flags duplicates for later merging● Auto-merging (use carefully!)● User-defined rule based picking of winners/losers● oldest/newest wins rules
Custom object matching/merging and cross object matching are also useful, but you might not need them
Manchester, NH 2019@NEDreamin #NEDreamin
Process for bulk merging
Do a first pass using exact matching followed by one or more fuzzy passes. Merge parent records first, child objects second (eg. merge accounts, then contacts). Save your searches/jobs/scenarios (you’ll need them again)
● Parent - exact pass 1● Child - exact pass 1● Parent - fuzzy pass 2● Child - fuzzy pass 2
Note: you may well have more contact duplicates after you merge accounts.
Why? Duplicate checking looks for duplicate contacts within accounts
Manchester, NH 2019@NEDreamin #NEDreamin
Tools for duplicate management - details
Switch to Google sheet here – URL for sheet is in resources
Manchester, NH 2019@NEDreamin #NEDreamin
Resources – Page 1Link to “Duplicate Tools” Google sheet:
https://bit.ly/30zTfcd
Manage Duplicates in Salesforce (Summer 2019 version) - 44 pages full of all the details on duplicate management including exactly how all the different flavors of fuzzy compares work.
http://resources.docs.salesforce.com/220/13/en-us/sfdc/pdf/salesforce_data_quality_duplicate_prevention.pdf
Youtube playlist – matching rules, duplicate rules, and duplicate reporting (Jay Delaune, Force501.org)
https://www.youtube.com/playlist?list=PLVkg_BRPOj3z_oCzzGi39JgO6SVQGP90C
Manchester, NH 2019@NEDreamin #NEDreamin
Resources – Page 2
NPSP merge/split households
https://powerofus.force.com/s/article/NPSP-Merge-or-Split-Households
Include Classic-style merging in Lightning (untested, may be unsuitable for NPSP)
https://forcepanda.wordpress.com/2019/07/30/merge-accounts-and-contact-in-lightning-like-classic/
Manchester, NH 2019@NEDreamin #NEDreamin
With special thanks to our Platinum sponsors!
Q & A