Best Of

Hi @stevenmckinnon - have you opened a support ticket for this to be investigated?

https://www.edq.com/documentation/contact-support/uk/

Just to close this one off, we discovered that there was a Data Studio issue preventing a scheduled workflow from export back to a table in a SQL Server system that used NTLM authentication.

We were able to work around the problem by using a setting in Data Studio to enable some as-yet-unreleased functionality. This functionality will be enabled by default from v2.1

Hi Patrick

I'm not going to pretend to understand the maths, but the prefix value gives higher ratings to strings that have a common prefix up to a prefix length of 4

The prefix weighting will be applied to the length of prefix that you supply and weight the results in preference of the strings with common prefixes.

Sorry, I confused myself I read AWS S3 and that was what i was looking at

I just tested this on v1.6.2 (should still hold true for v1.6.3

Hi @Carolyn

Find Duplicates works by standardising data and mapping the fields you supply to standardised fields - so you are correct in surmising that the fields get individually mapped. We have a tool that can help you to see the workings of this process called the Find Duplicates Workbench. You're local support team will be able to provide you with this tool and walk you through a few examples.

The good news is that all your requests are possible in Data Studio.

If you want to match on two fields combined, you could use the "Generic Field" option. Use a concatenate function to combine the First and Middle Name in to a single field in your dataset and then use the Generic Field mapping in Find Duplicates to match on that field.

Once you have mapped the field you'll need to modify your underlying rule set to include the new name rule. How you do that will depend on exactly what business rule you want to implement. Do you want to replace the existing individual field name match or add to it?

For your second question, i would again use the Find Duplicates Workbench tool to evaluate exactly why you are not getting good matches and to tweak the underlying individual rule set. Using this tool, it is quite straight forward to include an AND/OR rule for date of birth. It is possible to set up the rules so it treats an empty date of birth as an exact match, which would then revert the rules to using the name address fields.

It is a bit complex to include detailed steps to do this in this forum. It would be best if one of your local consultants can furnish you with a copy of the Find Duplicates Workbench and show you how to evaluate the rules and change them. I will contact your local team and try and arrange this but copying in @Ian Buckle @Sean Edmunds as an fyi.

To give you an idea of what the Workbench can show you:

You can view the Standardised fields for any two records in the Find Duplicates store:

You can then view how the rules evaluate those two records and why they do/don't match:

In this example you can see that the rule does not evaluate exact because the Forename rules are not evaluated as Exact.

You can then use the rules editor to tweak the rules:

You can also find detailed information on how Find Duplicates uses blocking keys and rules in our online docs:

https://www.edq.com/documentation/aperture-data-studio/find-duplicates-step/advanced-config/

Hi @Sami Laine in our conversations with the folks aat Collibra, we have determined that the co-implementation of Mulesoft together with Collibra and a platform like Data Studio represents a lot of implementation friction.

Accordingly one of the more recent implementations of Data Studio with Collibra, connects Data Studio to the Collibra platform directly and bypasses using Mulesoft altogether. @Ivan Ng can provide you with more details, you'll also find details of that integration on the Collibra marketplace accompanied by some overview slides .

The general structure to a rule is:

{Theme Name}.{Level}={ {Optional Group}.{Element or theme}.{Comparator}[{results}]

So, assuming you've configured your date column to have an element group name of "Dob", what you would be looking for is:

DOB.Exact={ #DOB.Date[ExactMatch] }
DOB.Close={ #DOB.Date.DateCompare[MonthYearMatch] }

The exact rule doesn't have a comparator because the ExactString comparator is the default if one isn't supplied.

ok, the approach I think you probably want is one where you derive a winning record with the final harmonize, and then join the data back to the data replacing the ID based on the clusterid from the harmonized record, this requires a branch fter the find duplicates but before the harmonize. Does that make sense?

this multiview illustrates the results in the top panel

the winning record in the middle

the original data in the bottom

@stevenmckinnon look for Standardize or GDQStandardize though it is possible that it may be on a different server from the one that Data Studio is installed on if you are using a remote matchstore configuration

https://www.edq.com/documentation/aperture-data-studio/help/#gdq-standardize-server-failed-to-start