Importing DataSets - Filtering Data

Rachael
Rachael Member
edited December 2023 in General

Hi,

I was wondering if there is currently anyway of limiting/filtering out data from an underlying table before importing it as a dataset? I have a number of transactional tables that I want to bring in to Aperture, however it would be great if I could limit the date range of the transactions before importing the table itself is upwards of 6 billion rows.

If there is no way of currently doing this, is it something that can be considered for a future release?

Thanks!

Rachael

Answers

  • Clinton Jones
    Clinton Jones Experian Elite

    HI Rachael

    It isn't possible right now but we do have plans to introduce something like this in the future, what kinds of database are you connecting to and what kind of data volumes are you looking at ?


    Clinton

  • Rachael
    Rachael Member

    Hi Clinton,


    Thanks - that's great news this is something which is being looked in to. The underlying tables have approximately 15-20 million records and it's connecting to a SQL database


    Rachael

  • Henry Simms
    Henry Simms Administrator
    Hi @Rachael , just letting you know that you can now specify a SQL query when loading data from a Database: https://docs.experianaperture.io/data-quality/aperture-data-studio-v2/get-started/configure-external-systems/#loaddatausingasqlquery

    This will allow you to filter on load, as well as pushing additional processing like grouping or joining down to the data source system.

    Available from V2 5.6 onwards: https://community.experianaperture.io/discussion/760/

Categories