Why would a snap shot drop all data in some columns when run as part of a workflow?
I have a workflow that acts as shell for several workflows that run in sequence (first cleaning data, the second evaluating that cleaning). I have set up a snapshot to capture the data after the cleaning to make sure everything is working OK.
When I run the workflow, this snapshot does not retain the data for the original columns and they are all blank (the transformed columns do get retained). If I go into the workflow and look at "show step results" for all stages up to and including the snapshot all the columns are retaining their data.
I have created a copy of the cleansing workflow to test this further, which directly inputs the data to this sub-workflow but is otherwise the same, and I have had the same results. I have run several different data sets through both variations and again the same result comes in each case where the snap shot has many blank columns. These are all in the attached dmx file.
Is this kind of behaviour and known bug, or could there be an issue with how we are generating the dataset through the snapshot function?
Many thanks! Ben
Answers
-
Hi @BAmos
Does you Take Snapshot step contain any unresolved schema changes? https://docs.experianaperture.io/data-quality/aperture-data-studio-v2/get-started/create-a-workflow/#take-snapshot_snapshotschemachanges
You would see this as a message on the step in the workflow.0
