Data Quality reporting over time
We have a set up here at Speedy where we have data quality split across multiple spaces. We pull all of the DQ validations out into a central space via snapshots and views to create a kind of DQ datamart which enables us to report DQ at a company level. One of the key issues we are challenged with is that users can occasionally change the name of a DQ validation rule to make it more clear or even just correct a spelling. The problem is this effectively creates a new DQ rule in the datamart. Ideally the name of the DQ rule should not be the primary key for a DQ rule - it should have an ID and then the validation should be a transactional FACT. IS anyone else facing this issue and have they found a way of dealing with it?
Comments
-
Hi Steven
I have seen a similar challenge previously. A user was building a chart using data being brought in from another system where the category/key can sometimes change. Depending on the frequency of these changes, you mention occasionally, a lookup list of the old value and new value could be a quick solution.
In your case, clone the rule name to another column using a Transform and in this new column replace any values in the lookup list. Use the new column in your charts and periodically review/update the lookup list.
===
Please share if you have ideas for a better solution or are running into any challenges documenting/reporting on rules
0 -
Had another thought that this might be over complicating it. Rather than duplicating the column (was thinking that to keep the old value, but it will still be available in the prior workflow step) you could just overwrite the existing column with a Transform lookup, which means the existing charts would not need updating.
0
