Data dropzone - additional columns

Rachael
Rachael Member
edited December 2023 in General

Hi!

I'm involved in a piece of work at the moment where we're using Aperture to map data in preparation for a data migration.

The pre-existing data can be imported using a database connection, however the templates which I'm mapping to have been provided via excel spreadsheets. The challenge I have is that there are newer versions of the templates, where some additional columns have been added. I know that the drop zone loads new data which has been provided in any files saved to the drop zone location, but when testing it, it didn't seem to recognise additional columns.

Any ideas/suggestions on a way to resolve this?

Thanks!

Best Answer

  • Sueann See
    Sueann See Experian Super Contributor
    Answer ✓

    @Rachael there may be several ways to do this.

    Configure new schema once using the UI

    You can upload the new data once using the UI and configure the new columns.

    At the datasets list, click on Options - Upload new data. With the UI, you will be prompted to configure the dataset when there are changes detected.

    image.png

    When there are additional columns found, you will see a message similar to this where col3 and col4 are the additional columns.

    image.png


    When you click on Configure and go to the Annotate columns step, you will see the changes as compared to your initial schema. (+) means the columns have been added. In addition, you can indicate whether any of the columns should be made optional or excluded in future.

    image.png

    Once configured, you can then use the dropzone for the next load.

    Reference: https://docs.experianaperture.io/data-quality/aperture-data-studio-v2/uncategorized/dataset-schema-changes/


    Use a map to target step within a workflow.

    Alternatively, you can create many datasets with different number of columns, and still map them to a target schema.

    Upload Dataset A with old schema.

    Upload Dataset B with new schema.

    Upload Dataset C as target schema (can be just header without data)

    In a workflow, connect the Dataset A to the map to target step selecting Dataset C as the target schema. Do the same for Dataset B.

    Reference: https://docs.experianaperture.io/data-quality/aperture-data-studio-v2/get-started/create-a-workflow/#map-to-target

Categories