\u00a0<\/p>\n
- \n
- Paribus 365™ Data Cleansing<\/li>\n
- Microsoft Dynamics\u00a0365 CRM<\/li>\n
- Infor CRM<\/li>\n
- Saleslogix<\/li>\n<\/ul>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
<\/p>\n
\u00a0<\/h2>\n
Just like any other project, cleaning up the duplicates in your data requires some work. How much time it will take is dependent upon the size of your data store, the size of your problem (how many duplicates you have), the speed of your infrastructure, the complexity of your decisions and the level of your knowledge.\u00a0 We hope to assist in the only one of these areas we have any control over \u2013 your knowledge.<\/p>\n
Note<\/strong> –\u00a0This article assumes the reader has read the included Paribus 365™ Data Cleansing documentation and understands the basic concepts of the building blocks of the Paribus 365™ Data Cleansing configuration, including Data Providers, Data Sets, Match Conditions, Match Sets and Match Sessions. In addition, you should be familiar with how to run a Paribus 365 Data Cleansing Session for your CRM system.<\/p>\n
Overview<\/h2>\n
The process of identifying and merging duplicate records using Paribus 365™ Data Cleansing (herein shortened to Paribus) can be distilled down to four steps:<\/p>\n
- \n
- Setup \u2013 This includes configuring Paribus 365 to point to the data source \u2013 or a filtered subset of the data source you\u2019re working with (i.e., your CRM system) and identifying what element(s) of data you wish to use as points of comparison, in what combinations, and at what matching thresholds (how similar data items need to be to be considered \u2018matches\u2019).\u00a0 All of this culminates in having what Paribus 365 calls a Match Session.<\/li>\n
- Running a Match Session \u2013 Having Paribus 365 execute the Match Session that you configured in Step 1.<\/li>\n
- Review \u2013 Having a person, or people, review the results of the matches Paribus 365 found to \u2018tell\u2019 Paribus:\n
- \n
- If you agree that members of the groups it found are actually duplicates (or not) and<\/li>\n
- Which of the group members in each group should be the Primary (surviving) record into which the data from the duplicate record(s) should be moved.<\/li>\n<\/ul>\n<\/li>\n
- Processing \u2013 Using the Paribus Plug-in built for your specific CRM system to execute the merging of all of the approved groups within the Match Session that you have reviewed.<\/li>\n<\/ol>\n
During Step 2 (Running a Match Session) and Step 4 (Processing), the computers are doing all the work, therefore our focus will be on the parameters configured in the Setup (Step 1), as this is where we can define the optimal configuration to minimize the time users spend in Review (Step 3).<\/p>\n
During setup, you are building a collection of re-usable building blocks.\u00a0 Although there\u2019s some work involved in getting these building blocks in place, this is time well spent so that you can minimize the time spent actually using Paribus to find and merge duplicates.<\/p>\n
Setup<\/h2>\n
There are a couple of key areas we can focus on to speed up the process of setting up Paribus for your use. When you\u2019re using Paribus with Saleslogix\/InforCRM, or Microsoft Dynamics CRM, the first thing you should do is import our pre-set Definitions we\u2019ve developed for that particular CRM system.\u00a0 This is the simplest way to get started with these CRM systems, as we\u2019ve completed much of the setup work for you. These Definitions are found in the …\\Application Support<\/em> folder of your Paribus kit.\u00a0 They\u2019re the files with the .pds extensions.<\/p>\n
The included Paribus 365 Definitions provide Data Sets for all of the data elements most commonly used for comparisons. They also include:<\/p>\n
- \n
- Pre-configured Conditions for secondary matching criteria<\/li>\n
- Match Set templates for quick creation of Match Sessions<\/li>\n
- Examples of commonly used Match Session configurations for identifying matching Accounts, Contacts and Leads within the standard CRM data configurations for these systems.<\/li>\n<\/ul>\n
So, to minimize the time needed to get Paribus 365 configured for you data, using the pre-set Definitions is clearly the best option as a starting point.<\/p>\n
For a better understanding of what we would do in the Setup to maximize efficiencies, let\u2019s start with a quick overview of the Review step.<\/p>\n
When reviewing results of a Match Session, users are answering two questions:<\/p>\n
- \n
- Are the members of this group of matches actually duplicates we would like to process?<\/li>\n
- If we agree that the records should be merged, which record should be the surviving record?
(The last section of this article contains special notes\u00a0related to the options for merging in Microsoft Dynamics CRM)<\/li>\n<\/ol>\nOK, those decisions sound pretty straightforward.\u00a0 So, how can we construct Match Sessions to give us the best results and spend the least amount of time in the Review process?\u00a0 First, let\u2019s talk about the size of your batch \u2013 yes, size does matter.<\/p>\n
When running the Match Session, Paribus is running query after query looking for duplicates. It\u2019s doing some heavy processing. If you\u2019re only looking at a few thousand records, the Match Session could take less than a minute or two to run.\u00a0 If you\u2019re looking at a few HUNDRED thousand records, it might take a few hours to run.\u00a0 OK \u2013 the computer\u2019s doing the work, so you can just walk away and do something else.\u00a0 But… once it\u2019s done, you\u2019ll have the potential of a HUGE list of Match Groups to review.\u00a0 There are some things to consider here:<\/p>\n
- \n
- If it will take days\/weeks to review the results, your reviewer(s) would likely feel like there\u2019s no light at the end of the tunnel.<\/li>\n
- There will have been lots of changes in the data between running the Match Session and performing the Cleansing Session.\u00a0 While Paribus can handle this just fine, there may be the chance that your decisions might have changed with changes in the data by that point.<\/li>\n
- The Merge will take a long time to process and we suggest that this step is run when there\u2019s minimal activity in the system to avoid any performance degradation or conflicts.<\/li>\n
- It will be a long time before your users will see any benefit from the work you\u2019re doing.<\/li>\n<\/ul>\n
As you can see, there\u2019s good reason to break things up into manageable-sized groups of records.\u00a0 There\u2019s no single way you\u2019ll find all the matches, so some thought into how you can logically break things into smaller groups can actually help speed up the project and show your users some immediate and growing improvement in quality.<\/p>\n
How should we break our data into smaller groups of records? That depends on your data and how you use it, but here\u2019s some ideas (click on the title for an example) to elicit ideas that could work for you.\u00a0 Note that some examples are from use with Infor CRM and others are for use with Microsoft Dynamics CRM, but in all cases, the same concepts apply:<\/p>\n
- \n
- Geographically \u2013 Filter by Country, State\/Province, ZIP\/Postal Code ranges, etc.<\/li>\n<\/ul>\n
![\"Paribus](\"https:\/\/www.paribuscloud.com\/help\/wp-content\/uploads\/2015\/05\/geographically-300x274.png\")
<\/a><\/p>\n- \n
- Management \u2013 Filter by Account Manager or record Owner, etc.<\/li>\n<\/ul>\n
![\"Paribus](\"https:\/\/www.paribuscloud.com\/help\/wp-content\/uploads\/2015\/05\/management-300x257.png\")
<\/a><\/p>\n- \n
- Type \u2013 Filter by Account Type, Contact Type, Status, SIC Code, etc.<\/li>\n<\/ul>\n
![\"Paribus](\"https:\/\/www.paribuscloud.com\/help\/wp-content\/uploads\/2015\/05\/account-type-300x258.png\")
<\/a><\/p>\n- \n
- Status \u2013 Filter by Active or Inactive. Or in more specifically, you could filter by whether an Account has Opportunities (or not) or Tickets\/Cases (or not), etc.<\/li>\n<\/ul>\n
![\"Paribus](\"https:\/\/www.paribuscloud.com\/help\/wp-content\/uploads\/2015\/05\/status-300x261.png\")
<\/a><\/p>\n- \n
- A mix of different criteria \u2013 Filter by a combination of items like Active in Florida or Customers who have Open Opportunities, etc.<\/li>\n<\/ul>\n
![\"Paribus](\"https:\/\/www.paribuscloud.com\/help\/wp-content\/uploads\/2015\/05\/mix-of-criteria-300x274.png\")
<\/a><\/p>\nYou get the idea.<\/p>\n
Note<\/strong> – It\u2019s generally OK to break Contacts up by the first letter of their Last Name (usually users at least get that single character right, even if they don\u2019t spell the name correctly), but DON\u2019T break Accounts up by the first letter of their name; you\u2019ll eliminate the ability to find matches between names like Container Store and The Container Store.<\/p>\n
So, how do you tell Paribus which records you want to look at?\u00a0 Filters.<\/p>\n
Paribus 365™ Data Cleansing Filters<\/h3>\n
A Match Session is based on a Data Set.\u00a0 This defines a) what data source you\u2019re looking at and b) which specific records within that data source you\u2019re comparing.\u00a0 Filters are a part of the Data Set definition and allow you to look at a subset of your database instead of the entire set of records.<\/p>\n
For instance, you\u2019ve created a Paribus Data Provider to point to your CRM system and you\u2019ve created a Match Session based on matching Account Names as your primary match criteria.\u00a0 Without a Filter, you would be telling Paribus to look at the entire set of Accounts in your CRM system for duplicates.\u00a0 In some cases, that\u2019s fine, but often you would want to either exclude certain records (e.g., inactive or hidden records) or you may want to create a group for a Match Session that is entirely made up of Customers and another group that are Non-Customers.<\/p>\n
With these two Filters, you can compare your Customers to your Non-Customers with the goal of merging any data contained in your Non-Customer records into your Customer records and de-activating or ultimately deleting the duplicates.<\/p>\n
One major benefit of this type of comparison is that you can then intentionally identify your Customers as being the Primary record (survivor) in each Group (records from Data Set A). This automatically answers Question 2 above, and eliminates the need for attention to this point in the Review, saving precious time.<\/p>\n
If your Match Session is based on matching criteria that is very strict (e.g., 99-100% on 2 or 3 items), then you should have a very high level of confidence in answering Question 1 above regarding which Groups are duplicates.\u00a0 For example, if you ran a Match Session looking for a 99% match on Company Name and a 99% match on Full Address, AND you agree that this is a good criteria for finding duplicates, then you should be fairly confident that your Session Results will only contain groups of actual duplicate Accounts.<\/p>\n
By combining a Filter comparison like Customer vs Non-Customer and the high level of similarity outlined above, in many cases you can eliminate the review entirely by clicking the Global Group Review button on the Session Review toolbar ( \u00a0
![\"toolbar\"](\"https:\/\/www.qgate.co.uk\/knowledge\/wp-content\/uploads\/2015\/05\/toolbar.png\")
<\/a>\u00a0 ) and choosing to mark every Group in the Session as Reviewed.<\/p>\nThe Process<\/h2>\n
So, let\u2019s put this together into a plan for cleaning up Accounts.<\/p>\n
- \n
- Set up some filters that logically group your Account records into defined groups that help you easily say that records from Group A will ALWAYS win over records from Group B.\u00a0 The above scenario of Customers vs Non-Customers is a good example.\u00a0 Other examples would be Accounts with Opportunities would win over Accounts without Opportunities or Accounts with an AccountID from your ERP system would always win over Accounts that don\u2019t have an AccountID.<\/li>\n<\/ol>\n
(Note<\/strong> \u2013 As outlined in the example above, you can create Filters based on multiple criteria.\u00a0 This is very relevant when you have a very large data set where you might want to separate the data even further.\u00a0 You could compare on something like Bob\u2019s Customers vs Bob\u2019s Non-Customers; i.e., filtering by both type and management.<\/p>\n
- \n
- Create a Match Session that compare at a very high match threshold (e.g., 99%) on Account Name and something like Full Address and apply the Customer Filter to A and the Non-Customer Filter to B and tell Paribus 365 to automatically nominate the member from A as Primary.\u00a0 There are sample Match Sessions just like this in the Paribus 365 Definitions you imported!<\/li>\n
- Run the Match Session.\u00a0 This will identify matching records that are so similar and oriented in the right direction already so that you can simply spot-check in the Review and mark all of them as \u201cReviewed\u201d.\u00a0 Then run the Cleansing Session for these approved results.\u00a0 This will merge all the duplicate data into the Account that was nominated as the Primary record and delete (or hide) the duplicate Account records.\u00a0 Your data is already starting to look better!<\/li>\n<\/ol>\n
You can now go back and run a similar session for a different Account Manager or geography or take the same configuration and simply lower the Match Score Threshold to allow for more \u2018fuzziness\u2019 in the comparison to catch more matches.<\/p>\n
Note<\/strong> – the lower you set the Match Score Threshold, the more care you\u2019ll need to take when reviewing, but take care \u2013 you\u2019ll also be introducing the possibility of Paribus finding matches that you disagree with and would not wish to merge.<\/p>\n
You\u2019re now on your way to cleaner data.\u00a0 The hope is that this will give you some ideas on how to logically set up your Filters to get some quick results that your users will appreciate.<\/p>\n
Options for merging in Dynamics CRM<\/h2>\n
There is a unique option within the Paribus 365 for Dynamics CRM plug-in that allows for the selection of the Primary record (surviving member in a Match Group) at the time of the merge. You can have Paribus automatically choose the Primary record in each Group by these criteria:<\/p>\n
For Account Merges:<\/p>\n
- \n
- Account most recently Created<\/li>\n
- Account most recently Modified<\/li>\n
- Oldest established Account<\/li>\n
- Account with most Contacts<\/li>\n
- Account with most recent History<\/li>\n<\/ul>\n
For Contact Merges:<\/p>\n
- \n
- Contact most recently Created<\/li>\n
- Contact most recently Modified<\/li>\n
- Oldest established Contact<\/li>\n
- Contact with most recent History<\/li>\n<\/ul>\n
For Lead Merges:<\/p>\n
- \n
- Lead most recently Created<\/li>\n
- Lead most recently Modified<\/li>\n
- Oldest established Lead<\/li>\n
- Lead with most recent History<\/li>\n<\/ul>\n
If your decision of which member of a Match Group can be determined by these comparative criteria, you might be able to entirely eliminate the need to take the time to choose a Primary record during the review. If you have a special need for some other comparative criteria, the Paribus SDK would allow you to develop your own Match Group evaluation specific to your needs (this would require .Net programming).<\/p>\n
If you have any questions about\u00a0running match sessions in Paribus 365™ Data Cleansing, please get in touch.<\/a>
<\/p>\n
- Set up some filters that logically group your Account records into defined groups that help you easily say that records from Group A will ALWAYS win over records from Group B.\u00a0 The above scenario of Customers vs Non-Customers is a good example.\u00a0 Other examples would be Accounts with Opportunities would win over Accounts without Opportunities or Accounts with an AccountID from your ERP system would always win over Accounts that don\u2019t have an AccountID.<\/li>\n<\/ol>\n
- A mix of different criteria \u2013 Filter by a combination of items like Active in Florida or Customers who have Open Opportunities, etc.<\/li>\n<\/ul>\n
- Status \u2013 Filter by Active or Inactive. Or in more specifically, you could filter by whether an Account has Opportunities (or not) or Tickets\/Cases (or not), etc.<\/li>\n<\/ul>\n
- Type \u2013 Filter by Account Type, Contact Type, Status, SIC Code, etc.<\/li>\n<\/ul>\n
- Management \u2013 Filter by Account Manager or record Owner, etc.<\/li>\n<\/ul>\n
- Geographically \u2013 Filter by Country, State\/Province, ZIP\/Postal Code ranges, etc.<\/li>\n<\/ul>\n
![\"Paribus](\"\/help\/wp-content\/uploads\/2017\/01\/geographically-300x274.png\")
<\/a><\/p>\n