Using 'Matches' functions to identify personal information (and automate the detection of this)

Danny Roden
Danny Roden Administrator
edited December 2023 in General

Following on from @Steve's discussion (here), I wanted to share some further thoughts on this subject as it's a topic that crops up quite regularly. If I have not got permissions to hold or process PII (personally identifiable information) then I'd want an automated means of flagging whether it is present in my data source(s) before I load them into the production processes/environments (e.g. data feeds from a supplier before loading into my analytics platform).


Types of PII

Given that PII data comes in lots of different shapes/forms, it's important to build a workflow that's right for your definition. Most PII can be identified in 2x ways:

  • PII Values - where we recognise the presence of certain values as constituting that the given cell/row contains PII data. Some examples of the types of values you would want to look out for include: country names, locations, salutations/titles, common forenames/surnames and job titles.
  • PII Formats - where the format of a specific set of characters can be used to determine the presence of PII data. Some examples of these include: postcode/zipcode, telephone numbers, passport number, credit card details, NHS no, NI no, cookies, IP address etc.


Building this in Aperture Data Studio

Through a relatively easy to build workflow, you can detect the presence of both of these types of PII using a couple of 'matches' functions and setup automation so that email alerts can be triggered to inform data stewards if this is detected. Alternatively you could also adapt the workflow so that it automatically strips any PII values/formats from the records.

My approach for building this on a sample data set involves the use of 2x lookup tables (one for PII values and another for PII formats/regular expressions):

image.png

Within the transform step, 4x new columns are created:

image.png
  • Conc List (a concatenated list of all the values in each of the columns I want to screen for PII dat)
image.png
  • PII Values Detected (the values detected using my PII values lookup)
image.png
  • PII Format Detected (the formats detected using my PII formats lookup)
image.png
  • Row Doesn't Contain PII (a true/false flag to indicate whether the record is suitable or not)
image.png


Email Alerts

Finally, the 'Fire Event' step can be used to setup email alerts (along with scheduling or triggering) so that when this data source is reprocessed on an automated basis I can be informed of the presence of any PII automatically, like so:

image.png

Comments

  • James Sidebotham
    James Sidebotham Member

    Is it possible to get the "Rows for PII Formats" file?

  • Danny Roden
    Danny Roden Administrator
    PII Formats.txt

    Here you go @James Sidebotham! Let me know how you get on. Feel free to adjust the expressions as you see fit. I've got a .dmxd that's compatible with Aperture Data Studio v2 for the above workflow too if you'd like access to this also:

    image.png

    Another approach to consider is building out the individual PII functions as individual functions (so they can be independently referenced within Data Studio) and then roll these up into your PII Detection function using 'If then Else' logic:

    image.png

  • James Sidebotham
    James Sidebotham Member

    Great! And yes I would be interested in your .dmxd - why reinvent the wheel! Thanks

Categories